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EXECUTIVE  SUMMARY 


Research  has  shown  that  greater  than  80%  of  incidents  in  high-risk  organizations  result 
from  human  error.  The  maritime  industry  is  no  exception.  It  is  estimated  that  75%  to  96% 
of  maritime  casualties  are  caused  by  human  error.  Although  training  and  feedback  are 
provided  on  the  technical  skills  and  knowledge  of  operators,  nontechnical  skills  are  rarely 
addressed.  Nontechnical  skills  are  the  cognitive,  social,  and  personal  resource  skills  that 
complement  technical  skills  and  contribute  to  safe  and  efficient  task  performance.  Several 
high-risk  industries  such  as  medicine,  nuclear  power  plants,  and  civilian  aviation  have 
created  behavioral  marker  systems  that  provide  a  mechanism  for  observing  and  assessing 
the  nontechnical  skills  of  operators.  However,  there  is  currently  no  system  available  to 
evaluate  nontechnical  skills  of  Surface  Warfare  Officers. 

It  is  not  possible  to  create  a  system  that  is  capable  of  assessing  the  nontechnical 
skills  required  for  every  watch  station  on  a  ship.  There  are  simply  too  many  individuals 
with  too  wide  a  range  of  responsibilities.  However,  as  the  Officer  of  the  Deck  is  the  most 
important  watch  station  on  surface  ships,  it  was  the  most  logical  place  to  begin 
addressing  nontechnical  skills. 

This  thesis  describes  the  development  of  the  Nontechnical  Skills  for  Officers  of 
the  Deck  (NTSOD)  rating  form.  The  purpose  of  the  NTSOD  rating  system  is  to  provide  a 
framework  for  evaluating  the  nontechnical  skills  of  U.S.  Navy  Surface  Warfare  OODs. 
The  NTSOD  system  supplements  the  current  OOD  qualification  process  by  providing  an 
objective  and  documented  assessment  of  the  nontechnical  skills  of  OOD  candidates. 

The  NTSOD  system  consists  of  four  categories  of  behavior  (leadership, 
communication,  situational  awareness,  and  decision  making),  which  are  subdivided  into 
10  more  specific,  and  observable,  elements  of  behavior.  The  framework  was  developed 
through  the  analysis  of  data  collected  from  qualified  OODs.  When  properly  utilized,  the 
authors  believe  that  the  NTSOD  system  can  produce  skilled  OODs,  and  increase  the 
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overall  mission  effectiveness  and  safety  of  the  surface  fleet  by  providing  feedback  on  a 
crucial  aspect  of  OOD  performance  that  is  currently  not  consistently  evaluated  across  the 
Surface  Fleet. 
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I.  INTRODUCTION 


A.  BACKGROUND 

Human  error  is  a  major  factor  in  accidents  in  the  transportation  and  industrial 
fields.  It  is  estimated  to  be  a  causal  factor  in  80%  of  mishaps  (Reason,  1990).  Surface 
warfare  is  no  exception.  Therefore,  there  is  a  need  to  develop  valid  methods  for 
addressing  human  error  in  the  Surface  Warfare  Officer  (SWO)  community.  These 
methods  will  raise  the  overall  awareness  of  SWOs.  Once  the  community  is  aware  of  the 
effects  of  human  error,  safeguards  can  be  established  to  limit  the  incidents. 

B.  OVERVIEW 

The  Officer  of  the  Deck  (OOD)  occupies  a  unique  position  in  a  naval  ship. 
Nowhere  in  military  or  civilian  life  is  there  a  parallel  to  the  range  and 
degree  of  responsibility  that  is  placed  in  the  hands  of  the  OOD.  As  direct 
representative  of  the  Captain,  he  or  she  acts  with  all  the  authority  of 
command  and,  next  to  the  Captain  and  the  Executive  Officer  (XO),  is  the 
most  important  person  on  the  ship.  (Stavridis  &  Girder,  2007,  p.  1) 

As  demonstrated  by  the  above  quote  from  the  Watch  Officer’s  Guide,  when 
underway,  the  OOD  is  the  most  imporiant  watch  station  onboard  United  States  Naval 
surface  vessels.  OPNAVINST  3120.32C  (2003)  established  the  basic  function  of  the 
watch.  “The  Officer  of  the  Deck  underway  has  been  designated  by  the  Commanding 
Officer  (CO)  to  be  in  charge  of  the  ship  including  its  safe  and  proper  operation  (pp.  4- 
18).” 

The  current  qualification  process  for  OODs  primarily  consists  of  the  completion 
of  the  Personnel  Qualification  Standards  (PQS)  and  an  oral  board  administered  by  the 
CO,  XO,  and  selected  department  heads.  The  PQS  for  OODs  consists  of  line  items  that 
qualified  OODs  verify  once  the  candidate  has  displayed  adequate  knowledge  of  that  item. 
These  line  items  primarily  concentrate  on  technical  skills  and  knowledge  such  as  visually 
determining  a  ship’s  target  angle  and  determining  compass  error  using  a  range.  An 
example  line  item  is,  ‘Set  up  a  stadimeter  and  use  it  to  determine  the  distance  to  a  ship.’ 
Once  the  candidate  has  sufficiently  displayed  the  ability  to  use  a  stadimeter,  a  qualified 
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OOD  will  sign  his  or  her  PQS,  thereby  indicating  that  he  or  she  is  qualified  in  that 
specific  area.  The  PQS  also  ensures  that  OODs  have  the  technical  knowledge  required  to 
respond  to  emergencies  and  infrequent  tasks  (e.g.,  mooring  to  a  buoy,  loss  of  steering). 
The  PQS  is  the  same  across  the  Surface  Fleet  regardless  of  the  type  of  ship.  Therefore,  it 
ensures  baseline  technical  skills  and  knowledge  for  all  OOD  candidates. 

While  technical  skills  and  knowledge  are  important  in  all  high-risk  organizations, 
including  the  military.  Admiral  Stavridis  and  Captain  Girrier  (2007)  identified  several 
other  characteristics  that  are  just  as  important  for  a  successful  OOD.  Stavridis  and  Girrier 
state  that  forehandedness,  vigilance,  judgment,  intuition,  and  leadership  are  traits  that  are 
just  as  valuable  to  OODs  as  technical  knowledge  and  skills.  These  traits  are  examples  of 
what  psychologists  have  termed  nontechnical  skills. 

Nontechnical  skills  are  the  “cognitive,  social,  and  personal  resource  skills  that 
complement  technical  skills,  and  contribute  to  safe  and  efficient  task  performance”  (Flin, 
O’Connor,  &  Crichton,  2008,  p.  1).  These  skills  are  not  mysterious  or  new  to  the  surface 
warfare  community.  Nontechnical  skills  include  skills  such  as  leadership, 
communication,  and  decision  making.  The  best  OODs  already  possess  these  skills  or 
develop  them  through  experience.  Nontechnical  skills  help  watch  standers  respond  to 
situations  in  an  efficient  and  timely  manner.  They  also  help  OODs  handle  situations  that 
the  current  qualification  process  does  not  address. 

Through  observation,  intuition,  and  experience,  CO’s  develop  an  impression  of 
the  nontechnical  skills  possessed  by  individuals  under  their  command.  The  CO,  however, 
rarely  has  time  to  observe  every  watch  stander  for  the  time  that  is  required  to  form  an 
accurate  assessment  of  the  watch  stander’s  competence.  It  is  also  difficult  to  translate 
intuition  or  a  ‘gut  feeling’  into  meaningful  data  for  making  a  decision  as  to  whether  the 
watch  stander  is  competent.  Unfortunately,  there  are  no  valid  and  reliable  tools  available 
to  assess,  and  provide  feedback  on,  the  nontechnical  skills  of  the  OOD  that  are  crucial  for 
safe  and  effective  performance. 
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C.  PROBLEM  STATEMENT 

Technical  advancements  and  increased  redundancies  in  ships  and  other  systems 
have  reduced  the  chances  of  mechanical  errors  at  sea  that  would  be  severe  enough  to 
cause  major  incidents.  As  a  result,  the  majority  of  groundings,  collisions,  and  other 
maritime  accidents  are  a  result  of  human  error.  A  United  States  Coast  Guard  (USCG) 
report  identified  human  error  as  a  contributing  factor  to  75-96%  of  maritime  casualties 
(Hetherington,  Flin,  &  Mearns,  2006).  Assessing  and  providing  training  and  feedback  on 
nontechnical  skills  could  reduce  these  percentages.  Providing  COs  with  a  mechanism  for 
evaluating  non-technical  skills  would  allow  them  to  identify  potential  shortfalls  in  the 
skills  of  OODs  and  train  them  more  effectively.  For  example,  a  deficiency  in  situational 
awareness  or  decision  making  could  be  addressed  and  corrected  during  qualification 
before  the  individual  ever  stands  a  watch.  Unfortunately,  despite  their  recognized 
importance,  commands  do  not  consistently  track  or  evaluate  nontechnical  skills  for  OOD 
candidates. 

D.  OBJECTIVES 

There  were  two  primary  purposes  to  this  thesis.  The  first  was  to  identify  the 
critical  nontechnical  skills  for  the  OOD  watch  station.  While  Stavridis  and  Girder  (2007) 
recognized  several  imporiant  nontechnical  skills,  the  traits  they  identified  are  difficult  to 
observe  and  assess. 

The  second  purpose  was  to  develop  a  valid  and  reliable  framework  of  non¬ 
technical  skills  for  OODs  so  that  they  so  that  they  can  be  observed  and  rated. 

E.  HUMAN  SYSTEMS  INTEGRATION  (HSI)  APPLICABILITY 

HSI  emphasizes  human  considerations  as  the  main  priority  to  reduce  life  cycle 
cost  and  optimize  system  performance.  It  is  a  multi-disciplinary  field  composed  of  eight 
basic  domains:  manpower,  personnel,  training,  human  factors  engineering,  system  safety, 
health  hazards,  survivability,  and  habitability.  This  thesis  focuses  on  addressing  three  of 
these  domains  as  applied  to  the  OOD  watch  station. 
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The  first  and  primary  domain  addressed  is  training.  Booher  (2003)  identified 
training  as  “the  requisite  knowledge,  skills,  and  abilities  needed  by  the  available 
personnel  to  operate  and  maintain  systems  under  operational  eonditions”  (p.  3).  This 
thesis  identifies  the  nontechnical  skills  that  every  OOD  should  possess.  The  development 
of  a  valid  and  reliable  system  to  evaluate  the  nontechnical  skills  of  OODs  will  improve 
the  training  of  OODs.  The  system  will  allow  COs  to  identity  areas  of  strength  and 
weakness  to  assist  potential  OODs  during  their  qualification  process. 

The  second  domain  targeted  with  this  thesis  is  personnel.  Personnel  is  described 
as  the  “aptitudes,  experiences,  and  other  human  characteristics  necessary  to  achieve 
optimal  system  performance”  (Booher,  2003,  p.  3).  This  thesis  identifies  key 
nontechnical  skills  that  can  improve  the  performance  of  OODs.  Since  every  individual,  to 
varying  degrees,  already  possesses  these  skills,  providing  a  framework  for  measuring 
them  could  have  impact  on  the  type  of  person  that  selected  to  stand  OOD  during  critical 
evolutions. 

The  last  domain  addressed  is  system  safety.  The  above  quote  from  Stravridis  and 
Girrier  demonstrates  the  immense  amount  of  responsibility  that  an  OOD  has  while  on 
watch.  An  ineffective  OOD  possesses  the  ability  to  cause  severe  damage  to  the  ship  and 
equipment  as  well  as  injury  to  the  ship’s  crew.  Improving  the  proficiency  of  the  watch 
standers  will  increase  the  overall  safety  of  the  ships. 

F.  SCOPE  AND  LIMITATIONS 

This  thesis  focused  exclusively  on  the  nontechnical  skills  critical  for  standing 
OOD  on  U.S.  Navy  surface  warships.  The  method  used  to  identify  the  skills  is  focused  on 
critical  or  stressful  moments  that  occur  while  standing  the  watch.  However,  the  system 
applies  to  the  full  spectrum  of  OOD  activities. 

While  some  categories  may  fit,  the  findings  should  not  be  applied  to  similar 
watches  on  other  platforms  such  as  submarines  or  merchant  ships  without  carrying  out 
research  to  assess  the  validity  of  the  system.  The  system  as  a  whole  also  was  not  designed 
to  address  other  watch  stations  such  as  Engineering  Officer  of  the  Watch  (EOOW)  or 
Tactical  Watch  Officer  (TAO). 
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II.  EVALUATING  NONTECHNICAL  SKILLS 


A.  MODELS  OF  HUMAN  ERROR 

James  Reason  (1990)  estimated  that  human  error  eauses  80%  of  aeeidents  in  high- 
risk  organizations.  He  identified  human  error  as  “a  generie  term  to  eneompass  all  those 
oeeasions  in  whieh  a  planned  sequenee  of  mental  or  physieal  aetivities  fails  to  aehieve  its 
intended  outeome,  and  when  these  failures  cannot  be  attributed  to  the  intervention  of 
some  chance  agency”  (p.  9). 

Based  upon  research  examining  failures  in  high  reliability  organizations,  as  well 
as  the  work  of  other  researcher  (e.g.,  Rasmussen,  1974),  Reason  (1997)  developed  an 
organizational  model  of  human  error  commonly  described  as  the  ‘Swiss  Cheese’  model. 
This  model  allows  for  the  identification  of  active  failures  and  latent  conditions  in  an 
organization.  Active  failures  are  unsafe  acts  that  have  a  direct  impact  on  the  safety  of  the 
system.  These  acts  are  normally  committed  by  the  system  operator  (e.g.,  pilot, 
maintenance  personnel,  OOD).  Latent  conditions  are  the  failures  or  oversights  that  allow 
or,  in  some  cases,  encourage  the  end  user  to  commit  the  unsafe  act  that  lead  to  the 
accident.  Latent  conditions  include  issues  such  as  poor  design,  failures  of  supervision, 
and  shortfalls  in  training.  Latent  conditions  arise  from  decisions  made  by  manufacturers, 
designers,  and  managers.  They  may  be  present  in  the  system  for  many  years  before  any 
mishaps  occur  (Reason,  1997).  Although  latent  conditions  are  important  contributors  to 
accidents  and  poor  performance,  the  focus  of  this  thesis  is  on  addressing  the  active 
failures  of  the  Surface  Navy,  and  more  specifically  the  OOD. 

Organizations  whose  performance  may  be  catastrophically  impacted  by  failures  in 
complex  human  technology  systems  are  known  as  high-risk  industries.  High  Reliability 
Organizations  (HROs)  are  the  sub-set  of  high-risk  organizations  that  succeed  in  avoiding 
catastrophes  in  high-risk  environments  (Roberts  &  Rousseau,  1989).  The  common  factor 
underlying  HROs  is  that,  while  a  failure  of  reliability  has  the  potential  for  death,  loss, 
damage  to  assets,  or  ecological  disaster,  these  organizations  have  developed  unique 
properties  that  enable  them  to  adapt  to  unexpected  events.  They  develop  this  ability  by 
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placing  increasing  importance  on  understanding  and  leveraging  the  role  of  the  human 
operator  (O’Connor  &  Cohn,  2010).  Carter-Trahan  (2009)  identified  U.S.  Navy  vessels 
as  HROs.  While  the  Navy  has  fewer  aeeidents  than  would  be  expeeted  due  to  the 
eomplex  environment  that  they  operate  in,  mishaps  do  still  oecur.  This  thesis  inereases 
the  understanding  of  the  human  operators,  in  this  ease  OOD,  by  identifying  the 
nonteehnieal  skills  that  are  important  to  safe  and  effeetive  operation. 

Flin,  O’Connor,  and  Crichton  (2008)  describe  non-technieal  skills  as  “the 
eognitive,  soeial,  and  personal  resource  skills  that  eomplement  teehnieal  skills,  and 
eontribute  to  safe  and  effieient  task  performanee”  (p.  1).  Situational  awareness, 
leadership,  teamwork,  and  deeision  making  are  all  examples  of  nonteehnieal  skills.  Over 
time,  effeetive  operators  naturally  develop  these  skills  in  order  to  funetion  at  high  levels 
of  performanee.  While  human  error  eannot  be  eliminated,  improvements  in  nonteehnieal 
skills  ean  identify  and  mitigate  potential  risks. 

B,  BEHAVIORAL  MARKER  SYSTEMS 

While  identifying  relevant  and  applieable  nonteehnieal  skills  is  erueial  to  safety, 
assessing  the  skills  is  also  very  important.  Flin  et  al.  (2008)  define  assessment  as  “the 
proeess  of  observing,  recording,  interpreting,  and  evaluating  individual  performanee, 
usually  against  a  standard  defined  by  a  professional  body,  a  eompany,  or  a  safety 
regulator”  (p.  269).  The  primary  teehnique  for  assessing  or  evaluating  nonteehnieal  skills 
is  through  the  use  behavioral  marker  systems. 

Klampfer  et  al.  (2001)  defined  behavioral  markers  as  “observable,  nonteehnieal 
behaviors  that  eontribute  to  superior  or  substandard  performanee  within  a  work 
environment”  (p.  10).  Behavioral  markers  derive  from  data  analysis  regarding 
performanee  that  eontributes  to  an  outeome,  either  sueeessful  or  unsuoeessful.  Behavioral 
markers  eould  be  used  in  any  domain,  but  they  are  most  frequently  used  in  industries  in 
whieh  safety  is  a  primary  eoneern.  Klampfer  et  al.  (2001)  identified  five  eharaeteristics  of 
a  good  behavioral  marker. 
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1.  It  should  describe  a  specific  and  observable  behavior  instead  of  an 
attitude  or  personality  trait.  The  operator’s  ability  to  delegate 
responsibilities  is  observable  and  a  good  indicator  of  leadership  skills. 
However,  the  operator’s  attitude  towards  leadership  is  not  observable, 
so  it  would  not  make  an  effective  behavioral  marker. 

2.  While  it  does  not  have  to  be  present  in  every  situation,  the  behavioral 
marker  should  have  a  causal  relationship  with  the  performance 
outcome.  In  other  words,  the  inability  for  the  operator  to  communicate 
should  result  in  poorer  performance  for  his  or  her  team. 

3.  It  needs  to  use  domain  specific  language.  A  behavioral  marker  that 
was  developed  for  the  maritime  shipping  industry  is  unlikely  to  be 
useful  in  a  nuclear  power  plant  control  room  due  to  maritime  specific 
language. 

4.  It  should  employ  simple  phraseology.  This  attribute  will  allow  the 
marker  to  be  used  and  understood  by  a  wider  range  of  individuals 
including  the  assessor  and  the  operator 

5.  It  should  describe  a  clear  concept. 

Organizations  can  use  behavioral  markers  for  multiple  purposes.  Behavioral 
markers  can  be  used  to  give  feedback  on  performance  to  individual  operators  and  teams. 
They  can  highlight  positive  and  negative  examples  of  performance.  The  can  also  provide 
a  common  terminology  for  training  and  debriefing.  The  most  common  use  for  behavioral 
markers  is  as  a  tool  to  measure  and  evaluate  performance  (Klampfer  et  ah,  2001). 

Behavioral  marker  systems  are  rating  tools  that  consist  of  multiple  behavioral 
markers  organized  in  a  manner  that  allows  an  assessor  to  assign  grades  to  each  skill.  The 
system  normally  consists  of  several  broad  category  nontechnical  skills  (leadership, 
communications,  etc.)  that  are  then  broken  down  into  smaller  elements  or  markers  that 
are  observable  and  relevant.  The  system  should  also  contain  a  rating  scale  that  allows  for 
consistent  assessment  across  different  categories. 

While  Klampfer  et  al.  (2001)  identified  the  seven  desirable  characteristics  for  a 
behavioral  marker,  Flin  et  al.  (2008)  identified  seven  properties  for  an  effective 
behavioral  marker  system  as  a  whole:  sensitivity,  reliability,  validity,  structure, 
transparency,  usability,  and  baselines. 

•  Sensitivity  refers  to  assessor’s  ability  to  distinguish  between  good  and  bad 
performance  based  upon  the  markers.  If  a  behavioral  marker  system 
includes  leadership  as  one  of  its  nontechnical  skills,  it  should  be  easy  to 
identify  what  markers  indicate  good  leadership  and  which  indicate  bad 
leadership. 
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•  Reliability  relates  to  the  stability  of  the  measurement.  In  other  words,  the 
same  action  should  be  given  the  same  rating  in  multiple  situations.  There 
are  three  primary  aspects  to  reliability:  test-retest,  internal  reliability,  and 
inter-rater  reliability.  Test-retest  refers  to  the  stability  over  time.  Raters 
should  make  the  same  judgment  on  the  same  incident  occurring  at  two 
different  times.  Internal  reliability  tests  the  level  of  agreement  between 
individual  markers  that  are  intended  to  measure  the  same  nontechnical 
skill.  For  example,  three  behavioral  markers  that  are  being  used  to 
measure  communication  skills  should  show  correlated  scores.  Inter-rater 
reliability  is  the  ability  of  different  raters  to  give  similar  actions  the  same 
score  (Flin  et  ah,  2008). 

•  Validity  is  the  extent  to  which  a  marker  actually  measures  the  nontechnical 
skill  for  which  it  was  designed  to  assess.  Behavioral  markers  should 
accurately  reflect  differences  in  performance.  There  are  two  basic  aspects 
of  validity:  face  validity  and  construct  validity.  Face  validity  is  how 
appropriate  the  construct  looks  to  the  user.  In  other  words,  do  the  markers 
for  decision  making  look  like  they  are  actually  measuring  decision 
making?  Construct  validity  is  whether  the  rating  system  is  actually 
measuring  what  it  claims.  In  other  words,  higher  scores  should  actually 
relate  to  safer  and  more  efficient  performance  (Flin  et  al.,  2008). 

•  Structure  refers  to  how  well  the  components  and  markers  are  organized. 
The  perfect  behavioral  marker  system  would  address  all  possible 
behaviors  and  there  would  be  no  overlap  between  the  categories. 
Unfortunately,  it  is  impossible  to  create  a  system  that  can  be  used  for 
every  situation.  There  always  will  be  some  overlap  between  the  categories 
(for  example,  situational  awareness  and  decision  making  are  intrinsically 
related).  The  goal,  however,  is  to  limit  the  overlap  and  increase  the  total 
coverage  as  much  as  possible  (Flin  et  ah,  2008). 

•  The  system  should  also  be  transparent  or  understandable  to  the  operators 
who  are  being  rated.  They  should  have  information  on  the  reliability  and 
validity  of  the  system,  and  they  should  know  the  criteria  against  which 
they  are  graded. 

•  The  behavioral  marker  system  needs  to  be  usable.  The  framework  should 
be  simple  and  easy  to  understand.  The  targeted  behaviors  should  be  easy 
to  observe,  and  it  should  not  significantly  increase  the  rater’s  workload. 
They  system  should  also  have  domain-specific  language  that  is  familiar  to 
both  the  assessor  and  operator. 

•  Baselines  refer  to  how  appropriate  the  performance  criterion  is  for  the 
experience  level  of  the  operators.  More  experienced  users  should  be  held 
to  a  higher  standard  than  trainees  or  beginners  (Flin  et  ah,  2008). 
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Flin  et  al.  (2008)  also  identified  three  signifieant  limitations  to  behavioral  marker 
systems.  First,  they  are  never  capable  of  capturing  every  possible  aspect  of  performance. 
The  variables  that  effect  performance  are  simply  too  wide  ranging  to  rate  with  one 
system.  The  second  limitation  is  that  there  may  be  limited  opportunities  to  observe  some 
behaviors.  Important,  but  infrequent,  capabilities  such  as  conflict  management  or 
intuitive  decision  making  may  not  occur  often  enough  to  rate  on  a  consistent  basis.  The 
final  limitation  is  the  inherent  limitations  of  human  observers.  While  human  raters  bring 
experience  and  skills  that  a  fully  automatic  assessor  cannot  match,  they  also  bring  their 
own  biases  and  perceptions. 

It  is  important  to  note  that  behavioral  marker  systems  need  to  be  domain-specific. 
O’Connor,  O’Dea,  and  Melton,  (2007)  illustrated  that  the  Human  Factors  Analysis  and 
Classification  System  (HFACS),  which  was  developed  for  military  aviation,  was  not 
appropriate  for  assessing  human  error  or  nontechnical  skills  for  U.S.  Navy  divers. 
O’Connor  et  al.  then  tried  to  apply  a  behavioral  marker  system  designed  for  the  offshore 
oil  industry.  Significant  changes  were  made  to  the  taxonomy  before  it  could  be 
effectively  applied  to  U.S.  Navy  diving. 

C.  DOMAINS  IN  WHICH  BEHAVIORAL  MARKER  SYSTEMS  HAVE 

BEEN  USED 

Behavioral  markers  are  expensive  to  develop  and  utilize  given  the  level  of 
training  required  for  users.  Consequently,  they  have  mainly  been  developed  for 
occupations  where  safety  is  prime  and  simulators  are  used  for  training  and  assessment 
(Flin  et  al,  2008).  The  following  paragraphs  will  briefly  discuss  behavioral  marker 
systems  that  have  been  developed  for  civil  aviation,  medicine,  and  research  that  has  been 
carried  out  in  the  civilian  maritime  industry. 

1.  CIVIL  AVIATION 

a.  University  of  Texas  Behavioral  Markers  System 

The  first  behavioral  marker  system  for  pilots  was  developed  as  part  of  the 
University  of  Texas  Human  Factors  Research  Project.  The  study  had  two  primary 
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purposes.  The  first  was  to  evaluate  the  effeetiveness  of  erew  resouree  management 
(CRM)  by  measuring  observable  behaviors.  The  seeond  goal  was  to  aid  in  the 
development  of  future  CRM  programs  (Klampfer  et  al.,  2001). 

After  the  projeet  produeed  the  original  set  of  behavioral  markers,  the  University 
of  Texas  began  eolleeting  systematie  data  on  all  aspeets  of  an  airline’s  operations.  The 
markers  were  incorporated  into  a  system  known  as  the  Line/LOS  Checklist.  As  the  use  of 
the  system  grew,  it  became  obvious  that  there  were  significant  differences  in  crew 
behaviors  during  flight.  The  researchers  then  modified  the  system  to  address  the  markers 
for  each  phase  of  flight.  The  current  system  is  shown  in  Figure  1  (Klampfer  et  al.,  2001). 
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Figure  1.  University  of  Texas  Behavioral  Markers  for  Airline  Pilots  (From 

Klampfer  et  al.,  2001) 

b.  Nontechnical  Skills  (NOTECHS)  of  Pilots 

In  1996,  legislation  created  a  need  for  a  generic  method  of  evaluating  the 
nontechnical  skills  of  pilots  that  would  be  applicable  across  Europe  (Kanki,  Helmreich,  & 
Anca,  2010).  Any  method  that  was  developed  had  to  be  respectful  to  cultural  and 
corporate  differences  and  usable  by  airline  instructors  and  examiners.  A  research  team 
consisting  of  pilots  and  psychologists  from  around  Europe  developed  the  NOTECHS 
system  to  assess  the  CRM  skills  of  individual  pilots. 


11 


The  researchers  began  by  examining  the  behavioral  marker  systems 
already  employed  by  a  number  of  large  European  airlines.  Each  system  had  problems  that 
did  not  allow  the  researchers  to  generalize  it  for  all  of  European  aviation.  The  systems 
were  either  too  general,  too  specific,  or  they  assessed  the  crew  as  a  whole  instead  of 
individual  pilots.  As  a  result,  the  researchers  developed  a  new  taxonomy  and  rating 
system  to  assess  pilot’s  nontechnical  skills  (Kanki,  Helmreich,  &  Anca,  2010). 

The  researchers  used  an  extensive  literature  review  and  an  examination  of 
existing  behavioral  marker  systems.  Airline  captains  with  considerable  experience  in 
using  behavioral  marker  systems  then  advised  on  the  final  design.  The  result  is  a 
NOTECHS  system  that  consists  of  four  nontechnical  skill  categories  (cooperation, 
leadership  and  managerial  skills,  situational  awareness,  and  decision  making)  with 
component  elements  or  markers  for  each.  Eigure  2  shows  the  NOTECHS  system 
(Klampfer  et  ah,  2001). 
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Figure  2.  The  NOTECHS  Behavioral  Marker  System  (From  Klampfer  et  ah,  2001) 


2.  MEDICINE 

a.  Nontechnical  Skills  (ANTS)  of  Anesthetists 

Fleteher  et  ah,  (2004)  developed  a  behavioral  marker  system  for  assessing 
the  nontechnical  skills  of  anesthesiologists.  The  performance  of  anesthesiologists  was 
often  recorded  using  video.  However,  there  was  no  valid  and  reliable  tool  to  rate  the 
nontechnical  performance  of  the  anesthesiologists.  Fletcher  et  al.  developed  the 
Anesthetists’  Nontechnical  Skills  (ANTS)  system  to  fill  that  void. 
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Fletcher  et  al.  (2004)  started  by  developing  a  prototype  taxonomy  using 
literature  review  and  examinations  of  existing  marker  systems.  They  started  with  eight 
nontechnical  skills;  leadership,  team  working,  communication,  task  management, 
situational  assessment,  situational  awareness,  decision  making,  and  personal  factors. 

They  then  conducted  29  interviews  using  the  critical  incident  technique 
(CIT)  method.  They  were  able  to  derive  116  statements  about  nontechnical  issues  from 
the  transcripts.  Fletcher  et  al.  (2004)  then  used  the  interview  statements,  observations  in 
theater,  and  anesthesia  incident  reports  to  trim  down  the  list  of  nontechnical  skills  and 
individual  elements  or  markers.  The  final  categories  were  team  management,  team 
working,  situational  awareness,  and  decision  making  (see  Figure  3). 


[y 

EltnrenLs 

Task  managcRiicn  t 

Plan uing  and  preparing 

Prioritising 

Providing  and  maintaining  standards 
Identifying  and  utilianc.  lEi^urocs 

Team  wortdnf. 

Co-ordinating  ajctivities  witii  ticajii  nrembers 
Ficlianging  information 

Using  antlaorily  and  asse rtiye ness 

Assessing  capabilities 

Supporting  o  tiers 

Situation  a\^;areness 

Gatliering  inforjiiation 

Recognising  and  undcr.danding 

A  ntid  |)ati  ng 

Dodsinn  making 

Identifying  options 

Ealancing  risks  and  selecting  optioru 

Re-eval  isati  ng 

Figure  3.  Nontechnical  Skills  and  Associated  Elements  for  the  ANTS  system  (From 

Fletcher  et  al.,  2004) 


b.  Nontechnical  Skills  (NOTSS)  of  Surgeons 

Building  on  the  work  carried  out  by  Fletcher  et  al.  (2004),  Yule,  Flin, 
Paterson-Brown,  Maran,  and  Rowley  (2006)  identified  a  training  gap  for  surgeons. 
Analysis  of  adverse  events  in  surgery  revealed  that  many  of  the  underlying  causes  were 
not  technical  failures.  Behaviors  such  as  communication  were  commonly  found  to  be  the 
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cause.  While  surgeons  needed  to  demonstrate  these  nonteehnieal  skills,  training  did  not 
address  them.  Yule  et  al.  (2006)  ereated  a  behavioral  marker  system  for  surgeons  entitled 
the  Nonteehnieal  Skills  for  Surgeons  (NOTSS)  system. 

The  researehers  interviewed  27  surgeons  utilizing  the  CIT  method.  They 
asked  the  surgeons  to  deseribe  eritieal  ineidents  in  whieh  they  were  involved.  The  authors 
used  the  interviews  to  identify  the  key  nonteehnieal  skills  for  surgeons.  They  were  able  to 
derive  a  list  of  150  nonteehnieal  skills  from  the  interview  transeripts.  Yule  et  ah,  (2006) 
used  the  list  to  develop  broad  eategories  and  assoeiated  elements.  A  group  of  eonsultant 
surgeons  then  redueed  the  strueture  into  a  taxonomy  eomprising  five  eategories 
(situational  awareness,  deeision  making,  eommunieation,  and  leadership)  and  12 
elements.  Figure  4  displays  the  NOTSS  system  (Flin,  Yule,  Paterson-Brown,  Rowley,  & 
Maran,  2006). 
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Figure  4.  The  NOTSS  System  (From  Flin  et  ah,  2006) 
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3. 


MARITIME  SHIPPING 


While  no  behavioral  marker  systems  have  been  developed  for  the  maritime 
industry,  the  Warsash  Maritime  Center  in  England  has  been  conducting  research  into  the 
feasibility  of  applying  behavioral  markers  to  the  maritime  industry  (Gatfield,  2005).  The 
research  has  three  primary  areas  of  focus.  The  first  is  understanding  how  behavioral 
markers  can  be  used  to  assess  the  crisis  management  capabilities  of  engineering  officers. 
The  second  is  developing  a  behavioral  marker  assessment  framework.  The  final  focus  is 
providing  the  maritime  community  with  an  understanding  of  how  a  behavioral  marker 
system  can  be  applied  to  the  industry. 

The  research  is  being  conducted  by  observing  exercise  scenarios  within 
simulators.  While  the  research  is  not  yet  complete,  several  behavioral  markers  have 
already  been  found  that  significantly  affect  the  overall  performance  of  engineers  in  the 
simulators.  Figure  5  shows  the  identified  behavioral  markers  and  the  associated 
nontechnical  skills  (Gatfield,  2005). 
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Behauoural  Marker 

Charactensation 

Ratio  of  the  degree  of  feedback  control  to 
the  degree  of  predictive  control 

Indication  of  the  level  of  situational 

awareness 

The  number  of  alternative  hypotheses  and 
actions  commimicated  to  team  members 

An  mdication  of  teamwork  and  the 
bmldms  of  a  shared  mental  model 

Level  of  satisficing  exhibited 

Considermg  only  as  many  alternatives  as 
needed  to  discover  one  that  satisfies. 

Communicatmg  m  a  way  that  shares  ones 
mental  model 

Buildmg.  maintaining  and  refimng  the 
accuracy  of  the  shared  mental  model  of  the 
team 

Relevance  and  Umelmess  of  unsohcited 
information  passed  between  team 
members. 

A  measme  of  the  degree  of  congruence 
between  the  mental  models  held  by 
mdisidual  team  members. 

Le\'el  of  anticipation  of  other  team 
members  needs. 

Indication  of  the  level  of  situational 

awareness 

Level  of  anticipation  of  future  actions  and 
task  requirements. 

Indication  of  the  level  of  situational 

awareness 

Focus  IS  too  much  on  the  reduction  of 
uncertamty 

Indication  of  a  tendency  towards  analytical 
decision-makmg.  and  away  from 
naturahshc  decision-making 

Tendency  to  focus  on  one  system  at  a  time, 
thereby  ignormg  the  dynamics  of  the 
complete  system. 

An  mdication  of  the  lack  of  a  situation 
overview 

Amount  of  sampling  behaGour  exhibited. 

An  mdication  of  the  updatmg  of  situational 
awareness  and  mental  model. 

Number  of  unfimshed  sentences. 

A  measure  of  uncertainn’ 

Delegation  of  work  tasks. 

A  measme  of  the  effective  use  of  all  team 
members,  and  the  allenarion  of  overload 

Patterns  of  movement 

Interpretation  of  patterns  of  movement  to 
determine  degree  of  situation  overview 

Figure  5.  Behavioral  Markers  Identified  for  Merchant  Marine  Engineering  Officers 

(From  Gatfield,  2005) 


D.  CONCLUSION 

Effective  nontechnical  skills  are  crucial  for  effective  performance  in  high  risk 
organizations.  Behavioral  markers  have  been  shown  to  be  a  valid  and  reliable  method  for 
providing  feedback  to  operators  on  nontechnical  skills  and  are  being  used  in  a  number  of 
high-risk  domains.  Given  the  prevalence  and  success  of  behavioral  marker  systems,  it  is 
suggested  that  they  may  be  an  effective  method  for  improving  nontechnical  skills  in  the 
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Surface  Navy.  The  next  ehapter  will  provide  evidenee  for  why  behavioral  markers  will 
benefit  performance  of  Surfaee  Navy  personnel.  The  chapter  also  will  explain  why  the 
OOD  watch  station  is  the  first  watch  for  which  behavioral  markers  should  be  developed. 
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III.  THE  NEED  FOR  A  BEHAVIORAL  MARKER  SYSTEM  FOR 

OODS 


A,  ACTIVE  FAILURES  IN  THE  MARITIME  INDUSTRY 

Dr.  Anita  Rothblum  (2000)  notes  that  the  material  eonditions  of  the  shipping 
industry  have  been  eonstantly  improving  over  the  last  40  years.  Improvements  in  hull 
design,  ship  stability,  propulsion,  and  navigational  aids  all  have  been  implemented  to 
reduee  easualties  and  inerease  effieieney.  Unfortunately,  despite  these  advaneements,  the 
maritime  easualty  rate  is  still  high.  Dr.  Rothblum  states  that  this  result  is  due  to  maritime 
shipping  being  a  people  system,  so  human  error  is  a  large  contributing  factor.  She 
estimates  that  human  error  causes  75-96%  or  maritime  casualties.  Studies  have  shown 
that  human  error  contributes  to  84-88%  of  tanker  accidents,  79%  of  towing  vessel 
groundings,  and  89-96%  of  collisions  (Rothblum,  2000). 

Dr.  Rothblum  also  reported  the  results  of  a  Dutch  study  of  100  maritime 
casualties.  The  study  found  multiple  causes  for  each  accident.  Of  the  100  incidents, 
human  error  contributed  to  96  of  them,  with  93  events  having  more  than  one  human 
error.  An  important  finding  of  the  study  was  the  fact  that  every  human  error  was  a 
necessary  condition  for  the  accident  to  occur.  In  other  words,  the  accidents  would  not 
have  happened  if  the  human  errors  did  not  occur  (Wagenaar  &  Groeneweg,  1987). 

In  order  to  illustrate  human  error  in  shipping.  Dr.  Rothblum  (2000)  breaks  down 
two  real-life  maritime  incidents.  The  first  accident  is  the  collision  between  the  motor 
vessel  (MA^)  SANTA  CRUZ  II  and  the  U.S.  Coast  Guard  Cutter  (USCGC) 
CUYAHOGA  that  resulted  in  the  death  of  1 1  Coast  Guardsmen.  The  vessels  were  able  to 
see  each  other  visually,  and  they  had  radar  returns.  There  were  no  mechanical 
malfunctions,  and  no  severe  environmental  conditions.  The  sole  cause  of  the  accident  was 
human  error  (Rothblum,  2000). 

The  captain  of  the  CUYAHOGA  made  the  first  mistake.  He  misread  the  light 
configuration  of  the  SANTA  CRUZ  II.  Therefore,  the  captain  did  not  have  an  accurate 
mental  picture  of  the  other  vessel’s  size  or  heading.  As  a  result,  the  Captain  ordered  a 


21 


turn  that  took  the  CUYAHOGA  directly  across  the  bow  of  the  other  vessel.  The  second 
fateful  error  was  the  failure  of  the  crew  to  notify  the  captain  of  his  mistake.  They  realized 
what  was  happening,  but  they  were  either  unwilling  or  afraid  to  question  the  captain’s 
decision.  Fatigue  and  excessive  workload  may  have  contributed  to  the  errors  committed 
by  both  the  captain  and  crew  because  the  cutter  was  undermanned  (Rothblum,  2000). 

The  second  incident  that  Dr.  Rothblum  examined  was  the  grounding  of  the 
TORREY  CANYON.  This  grounding  took  place  in  the  English  Channel  in  broad 
daylight  and  calm  weather.  It  resulted  in  the  spillage  of  100,000  tons  of  oil.  Eour  separate 
human  errors  contributed  to  the  accident.  The  first  two  combined  to  put  a  large  amount  of 
pressure  on  the  master  of  the  vessel  to  make  good  time.  The  master  believed  that  if  he  did 
not  make  his  intended  port  by  the  next  high  tide,  he  may  have  to  wait  up  to  five  days 
before  pulling  in.  He  also  needed  to  transfer  cargo  in  order  to  even  the  ship’s  draft.  He 
did  not  want  to  perform  the  task  underway  because  it  would  increase  the  chances  of  oil 
spilling  on  his  decks  and  he  did  not  want  to  have  the  appearance  of  a  messy  ship. 
Therefore,  the  master  was  in  a  hurry  to  pull  into  port. 

The  third  human  error  was  the  master’s  decision  to  go  through  the  Scilly  Islands 
instead  of  around  them  in  order  to  save  time.  Unfortunately,  the  captain  was  not  familiar 
with  the  route,  and  he  did  not  have  the  correct  charts  for  the  area. 

The  helmsman  made  the  final  mistake.  He  did  not  realize  that  the  ship  was  on 
autopilot,  so  when  he  attempted  to  execute  the  turning  order  given  by  the  master,  nothing 
happened.  By  the  time  the  mistake  was  discovered,  it  was  too  late  to  make  the  turn,  and 
the  vessel  ran  aground  (Rothblum,  2000). 

Dr.  Rothblum  reported  the  results  of  a  study  conducted  by  the  U.S.  Coast  Guard 
on  improving  safety  and  performance  through  human  factors  principles.  The  study  found 
that  the  three  largest  problems  were  fatigue,  inadequate  communication,  and  inadequate 
technical  knowledge  (Rothblum,  2000).  The  author  believes  that  improvements  to  these 
areas  could  contribute  to  the  prevention  of  future  maritime  casualties 
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David  Gatfield  (2005)  used  the  story  of  the  MA^  GREEN  EIEY  to  help  justify  the 
need  for  a  behavioral  marker  system  to  assess  the  competenee  of  crisis  management  in 
the  engine  control  room  of  merchant  vessels.  The  GREEN  EIEY  was  underway  in 
vicinity  of  the  Shetland  Islands  in  severe  weather.  At  one  point,  a  seawater  supply  line  in 
the  engine  room  broke  and  caused  significant  flooding.  The  engineers  gained  control  of 
the  flooding  just  as  the  engine  stopped.  The  chief  engineer  incorrectly  assumed  that  the 
flooding  caused  the  engine  to  stop.  Several  unsuccessful  attempts  were  made  to  restart 
the  engine  as  the  vessel  slowly  drifted  towards  the  island  of  Bressay.  The  Shetland 
Coastguard  dispatched  three  tugs,  a  lifeboat,  and  a  helicopter  to  assist  the  vessel.  In  the 
end,  the  GREEN  EIEY  ran  aground  and  broke  apart.  The  coastguard  managed  to  rescue 
the  entire  crew  except  for  one  sailor  who  was  swept  away  and  lost  at  sea  (Gatfield,  2005). 

The  investigation  showed  that  the  engine  failure  was  actually  due  to  the 
mechanical  over  speed  trip.  If  the  engineers  would  have  simply  reset  the  trip,  the  engine 
would  have  started  and  the  disaster  averted.  There  was  also  a  lack  of  situational 
awareness  and  communication  between  the  crew  and  the  rescuers  that  further 
complicated  the  situation.  While  this  situation  began  with  a  mechanical  failure,  human 
error  compounded  the  situation  and  led  to  the  actual  grounding. 

Hetherington,  Elin,  and  Mearns  (2006)  noted  that  technical  advances  in 
navigational  aids  have  reduced  the  occurrences  of  mechanical  errors  severe  enough  to 
cause  significant  incidents.  However,  these  advances  have  revealed  the  extent  to  which 
human  errors  are  responsible  for  collisions,  groundings,  and  other  maritime  accidents. 
The  authors  identified  three  main  levels  of  issues  that  lead  to  maritime  accidents:  design, 
personnel,  and  organizational.  While  all  three  are  contributing  factors,  the  personnel 
issues  are  considered  the  immediate  causes  of  most  incidents.  Personnel  issues  include 
stress,  situational  awareness,  and  communication  (see  Eigure  6  for  a  complete  list).  The 
authors  did  identify  organizational  and  management  issues  as  underlying  causes  to 
mishaps,  similar  to  Reason’s  Swiss  Cheese  model. 
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Figure  6.  An  Organizing  Framework  for  Human  Factors  That  Contribute  to 
Accidents  in  Shipping  (From  Hetherington  et  ah,  2006) 


B.  ACTIVE  FAILURES  IN  THE  U.S.  NAVY 

Although  the  U.S.  Navy  collects  a  large  amount  of  detailed  data  on  the  human 
factors  causes  of  aviation  mishaps,  the  same  is  not  true  in  the  surface  community.  Carter- 
Trahan  (2009)  found  that,  in  an  examination  of  1 1 1  major  mishaps  in  the  surface  warfare 
community  investigated  by  the  Naval  Safety  Center  from  1999  to  2009,  only  23%  of 
mishaps  were  attributed  to  human  factors  (the  remaining  were  attributed  to  material 
causes  12%,  and  unknown  causes  65%).  A  similar  finding  was  reported  in  an 
examination  263  U.S.  Navy  diving  mishap  reports  by  O’Connor,  O’Dea,  and  Melton 
(2007)  where  70%  of  the  mishaps  were  attributed  to  “unknown”  causes,  with  only  23% 
attributed  to  human  factors.  These  percentages  are  far  below  the  80%  of  mishaps  that  are 
attributed  to  human  error  in  naval  aviation  and  other  high-risk  environments.  Therefore,  it 
is  suggested  that  the  information  collected  on  surface  mishaps  may  be  an  underestimation 
of  the  extent  to  which  human  error  contributes  to  afloat  mishaps.  The  following 
paragraphs  will  review  a  few  case  studies  of  afloat  mishaps  in  which  human  error  was 
causal. 
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On  May  17,  1987,  the  USS  STARK  was  underway  in  the  Arabian  Gulf 
conducting  operations  to  protect  Kuwaiti  and  Saudi  oil  tankers.  That  evening,  two  Exocet 
missiles  struck  the  ship.  The  first  failed  to  detonate,  but  the  second  exploded  and  resulted 
in  the  death  of  37  sailors.  Miller  and  Shattuck  (2004)  examined  the  incident  with  respect 
to  the  Dynamic  Model  of  Situated  Cognition  (DMSC). 

The  sensors  onboard  the  STARK  were  limited  due  to  adverse  weather  conditions, 
so  they  relied  on  the  information  relayed  from  an  Air  Force  Airborne  Warning  and 
Control  System  (AW ACS)  aircraft.  The  Iraqi  aircraft  that  fired  the  missiles  was 
automatically  identified  as  friendly  because  it  was  a  French-made  Mirage  fighter.  At  one 
point,  the  Iraqi  fighter  made  a  sharp  turn  and  increased  speed  towards  the  STARK.  While 
the  watch  standers  perceived  this  information,  it  was  not  comprehended.  As  a  result,  the 
threat  was  not  recognized  (Miller  &  Shattuck,  2004). 

Seven  minutes  later,  the  Iraqi  aircraft  turned  directly  towards  the  ship. 
Unfortunately,  the  crew  did  not  detect  this  information.  By  the  time  the  aircraft  was 
actually  queried,  one  missile  had  already  been  fired,  and  the  other  quickly  followed.  No 
sensor  systems  tracked  the  firing,  and  the  first  person  to  realize  a  missile  was  enroute  was 
the  forward  lookout.  Unfortunately,  his  information  did  not  reach  the  combat  information 
center  (CIC)  quickly  enough,  and  the  missile  impacting  the  hull  was  the  first  that  most  of 
the  ship  knew  of  the  attack.  Failures  in  situational  awareness  and  unsafe  acts  by  the  crew 
ensured  that  the  ship  was  not  able  to  respond  to  the  attack.  As  a  result,  37  crewmembers 
lost  their  lives. 

On  July  3,  1988,  the  USS  VINCENNES  mistakenly  shot  down  Iranian  Flight  655. 
During  the  time  of  the  incident,  the  combat  information  center  was  extremely  busy.  The 
ship  was  tracking  two  separate  groups  of  Iranian  small  boats,  and  CIC  was  also  tracking 
the  VINCENNES’  helicopter,  which  was  airborne,  and  an  Iranian  military  P-3  aircraft 
also  in  the  area.  The  official  investigation  reports  that  the  ship  was  also  dealing  with  a 
fouled  gun  mount  and  was  maneuvering  extensively  in  an  effort  to  keep  the  other  gun 
unmasked  (Miller  &  Shattuck,  2004). 
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Flight  655  was  initially  identified  as  “unidentified  assumed  hostile.”  Shortly  after 
take-off,  the  Identifieation  Designation  Supervisor  reeeived  information  that  led  him  to 
believe  the  eontact  was  aetuahy  an  F-14  fighter  aireraft.  Although  one  officer  did 
mention  that  the  contact  could  be  a  commercial  aircraft,  it  was  officially  designated  as  an 
F-14. 

When  Flight  655  was  about  12-15  nautical  miles  away  from  the  VINCENNES, 
the  Tactical  Information  Coordinator  mistakenly  reported  that  the  contact  had  started  to 
descend.  These  mistakes  were  ah  human  errors  that  ultimately  led  to  the  downing  of 
Elight  655.  The  mistakes  were  caused  by  increased  stress,  decreased  situational 
awareness,  and  a  preconceived  notion  of  the  environment.  If  one  or  ah  of  these  human 
errors  had  been  avoided,  the  result  of  the  encounter  may  have  been  entirely  different 
(Miller  &  Shattuck,  2004). 

In  her  thesis,  Carter-Trahan  (2009)  conducted  a  case  study  of  the  USS  DWIGHT 
D.  EISENHOWER  ramming  the  Spanish  Bulk  Carrier  URDUEIZ  while  the  Spanish  ship 
was  at  anchor  in  the  Chesapeake  Bay.  Multiple  human  errors  contributed  to  the  collision. 
While  the  radar  navigation  team  was  able  to  obtain  fixes  on  the  other  vessel,  the  visual 
navigation  team  was  unable  to  obtain  a  fix  on  the  URDUEIZ.  This  lack  of  cohesion 
between  the  two  teams  created  a  sense  of  confusion,  especially  since  the  visual 
navigation  team  had  primary  responsibilities.  However,  the  Navigator  and  OOD 
continued  to  make  recommendations  despite  the  disconnect  in  information.  These 
recommendations  included  course  and  speed  changes  that  were  not  communicated  to  the 
CO  prior  to  or  after  execution.  These  mistakes,  combined  with  a  lack  of  communication 
from  the  harbor  pilot  and  the  watch  standers  onboard  URDUEIZ,  caused  the 
EISENHOWER  to  ram  the  other  vessel. 

While  data  from  the  Safety  Center  does  not  necessarily  reflect  a  high  amount  of 
human  error  in  Naval  incidents,  evidence  from  case  studies  suggest  otherwise.  Therefore, 
addressing  the  nontechnical  skills  of  the  SWO  community  is  likely  to  be  effective  in 
improving  safety  and  performance.  The  following  section  outlines  why  the  initial  focus 
should  rest  on  the  OOD  watch  station. 
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C.  RATIONALE  FOR  THE  FOCUS  ON  OOD 

OPNAVINST  3120.32C  establishes  the  basie  function,  duties,  responsibilities, 
and  authority  of  the  OOD.  The  function  of  the  OOD  is  simply  stated;  “The  OOD 
underway  has  been  designated  by  the  Commanding  Officer  to  be  in  charge  of  the  ship 
including  its  safe  and  proper  operations”  (pp.  4-18).  The  instruction  then  lists  the  18 
duties,  responsibilities,  and  authorities  that  the  watch  station  demands.  There  is  a  wide 
variety  of  actions  the  OOD  must  perform.  He  or  she  must  be  aware  of  the  tactical 
situation  and  geographic  factors  that  may  affect  safe  navigation.  The  OOD  shah  make  ah 
required  reports  to  the  CO.  He  or  she  must  supervise  the  personnel  on  watch  on  the 
bridge  and  carry  out  the  routine  of  the  ship  as  published  in  the  plan  of  the  day.  The  OOD 
needs  to  supervise  transmissions  on  ah  radio  circuits  and  conduct  on-the-job  training  for 
the  other  officers  and  enlisted  personnel  on  the  bridge.  This  list  is  just  a  small  sample  of 
the  immense  and  varying  responsibilities  that  are  inherent  to  the  OOD  watch  station  (for 
the  complete  list  of  duties,  please  refer  to  OPNAVINST  3120.32C,  pages  4-18  through 
4-20). 

The  OOD  reports  directly  to  the  CO  for  the  general  operation  of  the  ship  and  to 
the  Executive  Officer  (XO)  for  the  ship’s  routine.  There  are  also  certain  situations  where 
the  OOD  may  report  to  the  Navigator.  OPNAVINST  3120.32C  identifies  13  separate 
watch  stations  that  report  directly  to  the  OOD.  Figure  7  shows  a  simplified  version  of  the 
underway  watch  structure. 


27 


Admiral  Stavridis  and  Captain  Girrier  (2007)  discuss  the  importance  of  the  OOD 
watch  station.  “Nowhere  in  military  or  eivilian  life  is  there  a  parallel  to  the  range  and 
degree  of  responsibility  that  is  placed  in  the  hands  of  the  OOD  (p.  1).”  Stavridis  and 
Girrier  feel  that  the  duties  and  responsibilities  delineated  in  OPNAVINST  3120.32C  are 
just  the  beginning.  Special  missions  and  circumstances  can  add  more  responsibilities,  but 
they  ean  never  be  redueed  from  what  is  stated  in  the  instruetion. 
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The  OOD  is  accountable  to  the  CO  for  every  event  that  occurs  during  his  or  her 
watch.  “As  the  captain’s  direct  representative,  the  OOD  is  the  only  person  on  board  who 
can  make  decisions  that  affect  the  safety  of  the  ship  and  the  lives  of  her  crew  (Stavridis  & 
Girrier,  2007,  p.  3).”  The  CO  cannot  possibly  be  on  the  bridge  at  all  times.  In  his  absence 
the  OOD  works  under  his  authority  and  speaks  with  his  or  her  voice. 

The  OOD  is  the  most  important  watch  station  onboard  U.S.  Navy  surface  ships. 
The  responsibilities  of  the  watch  are  wide  ranging,  and  his  or  her  decisions  affect  the 
entire  crew.  Therefore,  observing  and  assessing  the  nontechnical  skills  of  the  OOD  may 
improve  the  overall  safety  and  effectiveness  of  the  ship  as  a  whole.  While  many  different 
watch  stations  and  positions  could  be  analyzed,  addressing  the  nontechnical  skills  of  the 
OOD  will  go  the  furthest  towards  addressing  human  error  in  the  Surface  Navy. 

D,  CONCLUSION 

Chapter  II  illustrated  how  a  number  of  high-risk  organizations  have  already 
recognized  the  need  to  address  active  failures  and  nontechnical  skills  of  operators  by 
using  behavioral  marker  systems.  This  chapter  focused  on  establishing  the  need  to 
address  failures  of  nontechnical  skills  on  U.S.  Navy  surface  ships.  Unfortunately,  it  is  not 
possible  to  design  a  system  that  will  assess  the  nontechnical  skills  of  an  entire  ship’s 
crew,  nor  can  one  thesis  develop  a  framework  of  nontechnical  skills  for  every  watch 
station.  Therefore,  in  the  following  chapter  a  study  is  carried  out  to  develop  a  behavioral 
marker  system  for  the  OOD. 
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IV.  METHOD 


A.  INTRODUCTION 

The  literature  provides  three  primary  methods  for  identifying  nonteehnical  skills: 
event-based  analyses,  questioning  teehniques,  and  observational  teehniques  (Flin  et  ah, 
2008).  Event-based  analyses  consist  of  examining  accident/near  miss  reports  and/or 
examining  information  collected  for  confidential  reporting  systems.  Accident  and  near- 
miss  assessments  entail  analysis  of  past  mishaps  that  have  been  recorded  by  the 
organization.  The  advantage  of  this  method  is  that  an  actual  incident  has  occurred  that 
was  severe  enough  to  warrant  investigation.  Therefore,  actual  examples  may  be  found 
where  certain  nontechnical  skills  were  lacking  and  caused  the  accident.  The  disadvantage 
of  accident  analysis  is  that  the  reports  may  be  incomplete  and  only  represent  part  of  the 
overall  incident  (Flin  et  ah,  2008). 

Confidential  reporting  systems  collect  information  from  workers  about  their 
mistakes  or  safety  concerns  that  they  would  not  normally  be  report.  When  this  data  are 
successfully  fed  back  to  leadership  and  management,  safety  improvements  can  be  made. 
Unfortunately,  individuals  may  be  reluctant  to  report  their  own  mistakes  or  concerns  even 
when  the  system  is  confidential.  However,  organizations  such  as  NASA  and  British 
Airways  have  successfully  used  confidential  reporting  systems  to  obtain  information  on 
unsafe  events  (O’Connor  &  O’Dea,  2007). 

The  second  major  system  for  identifying  nontechnical  skills  is  questioning 
techniques.  This  methodology  can  be  divided  into  interviews,  focus  groups,  and 
questionnaires.  There  are  a  three  main  interview  types:  structured,  semi-structured,  and 
unstructured.  Interviews  are  advantageous  because  the  interviewer  is  able  to  ask  about  the 
information  that  is  relevant  to  him  or  her  instead  of  hoping  that  an  accident  report 
contains  pertinent  data.  The  main  disadvantage  of  interview  techniques,  however,  is  that 
the  interviewee  is  only  going  to  provide  the  information  that  he  or  she  is  comfortable 
giving  and  is  able  to  verbalize.  If  the  person  is  reluctant  to  incriminate  themselves  or 
others,  the  interview  may  not  provide  accurate  or  relevant  information. 
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Focus  groups  involve  a  group  of  people  providing  feedback  on  a  product  or  topic 
(Flin  et  al.,  2008).  The  interaction  between  group  members  creates  higher  levels  of 
discussion  on  specific  topics.  The  advantage  is  that  concentrated  amounts  of  information 
ean  be  gathered,  especially  on  events  that  are  not  direetly  observable.  The  disadvantage  is 
that  the  moderator  or  a  powerful  group  member  can  control  the  diseussion  to  the  point 
that  the  data  are  no  longer  accurate. 

Organizations  can  use  questionnaires  to  collect  specific  information  on 
nontechnical  skills  such  as  leadership,  teamwork,  and  situational  awareness.  They  have 
the  advantage  of  being  inexpensive  and  more  efficient  when  eolleeting  data  from  large 
groups  of  people.  Unfortunately,  there  are  limitations  to  the  amount  and  type  of 
information  that  ean  be  collected. 

The  third  technique  that  has  been  used  to  identify  nonteehnical  skills  are 
observations.  Researehers  can  use  observations  to  eollect  information  about  the 
workplace  settings  or  a  potential  problem  (Flin  et  ah,  2008).  Observations  ean  be  divided 
into  three  types:  direct,  participant,  and  remote.  The  advantage  of  observation  is  that  the 
researeher  is  able  to  witness  the  information  first-hand.  Unfortunately,  there  is  no 
guarantee  that  the  desired  aetion  will  happen,  and  it  is  possible  that  the  researeher  ean 
alter  the  behaviors  of  the  operators  through  his  or  her  presenee. 

After  considering  these  different  methodologies,  the  author  deeided  to  use  a 
combination  of  a  literature  review,  a  foeus  group,  and  interviews  to  identify  the 
nonteehnical  skills  of  OODs.  There  are  multiple  reasons  for  this  approaeh.  First,  as 
discussed  in  the  previous  chapter,  the  eurrent  mishap  classification  system  does  not 
adequately  capture  human  error  (Carter-Trahan,  2009)  and  no  near-miss,  or  confidential 
reporting  system  exists.  Sinee  this  thesis  focused  on  the  performance  of  OODs  during 
stressful  situations,  observational  methods  also  were  not  appropriate.  Due  to  the  rarity  of 
sueh  events  in  the  surface  navy,  there  is  no  guarantee  that  a  researcher  would  have  been 
able  to  observe  the  desired  situation.  However,  the  Naval  Postgraduate  School  contains  a 
wealth  of  knowledgeable  and  experieneed  OODs.  The  author  took  advantage  of  the 
presenee  of  these  experienced  OODs  to  lead  a  foeus  group  and  interview  individuals 

about  situations  they  had  eneountered  in  the  past  while  standing  the  watch  as  an  OOD. 
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There  is  precedent  for  this  method  as  the  literature  review  provided  multiple  examples  of 
researchers  using  subject  matter  experts  to  help  create  and  refine  their  behavioral  marker 
systems  (Fletcher  et  ah,  2004;  Yule  et  al,  2007). 

B,  LITERATURE  REVIEW 

1,  Purpose 

The  purpose  of  the  literature  review  was  to  create  an  exhaustive  list  of  categories 
of  nontechnical  skills  that  have  been  shown  as  necessary  for  effective  performance  in 
high-risk  domains,  with  a  particular  focus  on  any  research  that  had  been  carried  out  in  the 
maritime  industry.  A  comprehensive  review  provided  a  framework  to  begin  production  of 
the  initial  taxonomy  of  nontechnical  skills.  Textbooks,  articles,  and  reports  about  other 
behavioral  marker  systems  established  a  starting  point  for  identifying  the  nontechnical 
skills  that  are  applicable  to  the  OOD  watch  station. 

2,  Method 

The  author  consulted  documentation  to  compile  a  list  of  nontechnical  skills  used 
in  other  fields.  The  only  reference  to  behavioral  markers  in  the  maritime  industry  was  by 
Gatfield  (2005).  He  used  simulator  observations  to  develop  a  set  of  markers  for  assessing 
the  competencies  of  merchant  marine  engineering  officers  during  crisis  management. 
However,  the  behavioral  markers  were  not  developed  at  the  time  of  publication  (nor  are 
they  discussed  in  any  subsequent  publications).  Given  the  dearth  of  research  on 
nontechnical  skills  in  the  maritime  industry,  four  behavioral  marker  systems  that  were 
available  in  the  literature  (LLC,  NOTECHS,  ANTS,  and  NOTSS)  were  examined.  Also 
included  were  the  nontechnical  skills  identified  by  Stavridis  and  Girrier  (2007),  the  skills 
discussed  by  Flin  et  al.  (2008),  and  the  relevant  personnel  issues  identified  by 
Hetherington  et  al  (2006).  This  literature  review  resulted  in  the  identification  of  17 
categories  of  nontechnical  skills. 
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3. 


Results 


The  literature  review  created  the  17  nontechnical  skills  shown  in  Table  1. 


•  Situational  awareness 

•  Judgment 

•  Decision  making 

•  Intuition/experience 

•  Communication 

•  Energy 

•  Team  working 

•  Co-operation 

•  Eeadership 

•  Management  skills 

•  Managing  stress 

•  Task  management 

•  Coping  with  fatigue 

•  Workload  management 

•  Eorehandedness 

•  Assertiveness 

•  Vigilance 

Table  1.  Initial  List  of  Nontechnical  Skills 


4.  Discussion 

The  author  used  the  literature  review  to  lay  a  foundation  of  nontechnical  skills 
that  could  be  considered  applicable  to  the  OOD  watch  station.  The  list  included  many 
skills  that  were  overlapping  or  redundant,  but  it  provided  an  excellent  starting  point  for 
identifying  which  of  the  skills  were  important  to  the  OOD. 

C.  FOCUS  GROUP 

1,  Purpose 

Several  of  the  nontechnical  skills  were  highly  correlated,  redundant,  or  not 
applicable  to  the  OOD  watch  station.  Therefore,  the  challenge  was  to  identify  which  of 
the  nontechnical  skills  in  Table  1  were  appropriate  for  assessing  OODs.  Once  this  had 
been  carried  out,  the  goal  was  for  the  SMEs  to  then  agree  upon  an  initial  taxonomy  of 
skills  sub-divided  into  specific  elements  (the  same  structure  as  NOTECHS). 

2.  Method 

a.  Participants 

The  focus  group  consisted  of  four  U.S.  Navy  junior  officers  led  by  an 
experienced  OOD  (the  author).  Each  participant  was  a  Surface  Warfare  Officer  (SWO) 
attending  the  Naval  Postgraduate  School.  All  of  the  participants  had  been  qualified  as  an 
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OOD  for  a  minimum  of  a  year.  They  had  all  stood  the  wateh  numerous  times  and  spent 
time  training  others  to  do  the  same.  They  were  also  Human  Systems  Integration  Masters’ 
students,  and  had  attended  elasses  on  human  factors,  safety,  individual  differences,  and 
team  working. 


b.  Procedure 

The  author  presented  each  participant  with  the  comprehensive  list  of 
nontechnical  skills,  definitions  of  each  skill,  and  examples  of  behavioral  marker  systems. 
The  participants  were  asked  to  familiarize  themselves  with  the  material  prior  to  meeting. 
The  author  explained  to  the  participants  that  the  goal  was  to  trim  the  initial  list  of  skills  to 
make  it  applicable  to  the  OOD  watch  station.  The  secondary  goal  was  to  identify 
individual  elements  for  each  skill.  The  author  encouraged  open  discussion  among  the 
participants  and  answered  any  questions  that  arose. 

The  group  began  by  eliminating  nontechnical  skills  that  they  felt  were 
either  redundant  or  not  applicable.  For  example,  the  participants  removed  judgment 
because  they  felt  it  was  included  under  the  decision  making  category.  Co-operation  was 
eliminated  because  the  focus  group  felt  that  is  was  not  applicable  since  the  study  was 
addressing  individual  instead  of  team  skills.  The  participants  removed  other  skills  such  as 
forehandedness  because  the  meanings  were  ambiguous  and  difficult  to  rate. 

Once  the  focus  group  had  identified  a  list  of  nontechnical  skills,  they  were 
asked  to  use  these  to  formulate  a  taxonomy  of  categories  and  associated  elements.  The 
participants  were  encouraged  to  draw  from  their  experience  as  an  OOD  to  identify 
specific  behavioral  markers  skills.  Following  the  discussion,  the  group  reached  a 
consensus  on  five  categories,  each  with  two  or  three  corresponding  elements.  The 
taxonomy  that  resulted  from  the  focus  group  is  shown  in  Table  2. 

3.  Results 

The  focus  group  trimmed  the  list  of  17  nontechnical  skills  down  to  five: 
leadership,  decision  making,  situational  awareness,  communications,  and  stress 
management.  The  participants  also  identified  two  or  three  elements  for  each  skill. 
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Because  this  is  only  the  initial  taxonomy  and  alterations  are  expected,  definitions  and 
descriptions  of  the  categories  will  not  be  provided  here.  Please  see  the  results  section 
following  the  inter-rater  reliability  testing  for  descriptions  of  the  elements  in  the  final 
taxonomy.  See  Table  2  for  the  complete  initial  taxonomy  of  skills  and  elements. 


Nontechnical  Skill  Category 

Elements 

Leadership 

Establishing  Authority 

Managing  Workload 

Maintaining  the  Standards  of  the  Watch 

Decision  Making 

Defining  the  Problem 

Generating  Possible  Solutions 

Implementing  Best  Solution 

Situational  Awareness 

Actively  Gathering  Information 

Responding  to  Changes  in  Information 

Anticipating  Future  Events 

Communication 

Selecting  Correct  Medium 

Sending  Information  Clearly  and  Concisely 

Effectively  Receiving  Information 

Managing  Stress 

Maintaining  Concentration 

Coping  with  Stressors 

Table  2.  Initial  OOD  Nontechnical  Skills  Taxonomy 


4.  Discussion 

The  author  took  advantage  of  the  abundance  of  experience  and  knowledge  located 
at  the  Naval  Postgraduate  School  to  assist  in  the  identification  of  the  applicable 
nontechnical  skills.  The  focus  group  took  the  initial  list  of  skills  established  from  the 
literature  review  and  specifically  refined  it  for  the  OOD.  The  participants  also  identified 
the  individual  elements  that  apply  to  each  skill  category.  This  method  produced  the  initial 
OOD  nontechnical  skills  taxonomy  shown  in  Table  2. 

D.  CRITICAL  INCIDENT  INTERVIEWS 
1,  Purpose 

While  the  initial  taxonomy  appeared  to  be  relevant  and  applicable  to  the  OOD 

watch  station,  the  author  decided  to  test  its  validity  using  real-life  examples.  The  author 
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interviewed  qualified  OODs  regarding  their  experienees  while  standing  the  wateh  during 
stressful  situations.  These  interviews  provided  the  author  with  data  from  real  situations. 
This  data  was  used  to  eonduet  inter-rater  reliability  testing  to  identify  potential  issues 
with  the  taxonomy.  Seetion  E  of  this  ehapter  diseusses  this  testing. 

2.  Method 

a.  Participants 

Eight  individuals  volunteered  to  partieipate  in  the  interview  proeess.  Eaeh 
person  was  a  Surfaee  Warfare  Offieer  attending  Naval  Postgraduate  Sehool.  Eaeh 
partieipant  had  qualified  as  OOD  at  a  previous  eommand  and  spent  signifieant  time 
standing  the  wateh  and  training  others.  Seven  of  the  volunteers  had  reeently  eompleted 
their  division  offieer  tours  while  the  remaining  individual  was  on  the  shore  duty 
following  his  department  head  tours. 

b.  Procedure 

The  author  interviewed  eaeh  volunteer  separately.  Eaeh  partieipant  was 
asked  to  reeount  a  stressful  or  otherwise  memorable  series  of  events  that  took  plaee  while 
he  or  she  was  standing  OOD.  The  author  eneouraged  eaeh  partieipant  to  deseribe  as  many 
experienees  as  they  were  eomfortable  providing.  The  interviews  lasted  between  25  and 
60  minutes  with  an  average  of  45  minutes.  The  author  utilized  the  Critieal  Ineident 
Teehnique  (CIT)  to  eonduet  the  interviews. 

The  CIT  was  first  developed  and  used  by  Elanagan  (1954)  to  aid  in  pilot 
seleetion.  However,  the  CIT  teehnique  has  been  widely  used  in  studies  of  human  error 
and  safety  (Kirwan  &  Ainsworth,  1992).  The  CIT  enables  the  researeher  to  identify  the 
(often  taeit)  knowledge  of  skills  and  expertise  possessed  by  respondents  by  asking  them 
to  deseribe  a  ehallenging  ineident.  It  goes  beyond  proeedural  knowledge  by  probing  the 
behavioral  aspeets  of  experienee  (O’Connor  et  al,  2008).  The  CIT  interviews  were 
eondueted  in  four  sweeps,  defined  below,  by  a  qualified  OOD  knowledgeable  about 
human  faetors.  Eaeh  sweep  was  designed  to  jog  the  partieipant’ s  memory  and  get  as 
mueh  applieable  and  aeeurate  information  as  possible. 
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Sweep  1  -  Prompting  the  interviewee  to  identify  a  relevant  ineident:  Each 
participant  was  asked  to  select  and  describe  an  event  that  occurred  when  they  were 
standing  watch  as  the  OOD  which  they  found  to  be  particularly  challenging.  They  were 
asked  to  describe  the  event  from  their  own  perspective  and  to  describe  it  in  detail,  stage 
by  stage,  as  it  developed. 

Sweep  2  -  Filling  in  gaps  in  the  incident:  The  interviewer  repeated  the 
reported  incident  back  to  the  respondent,  in  order  to  check  understanding.  The  respondent 
was  told  they  should  correct  any  mistakes  in  the  account  or  add  any  information  that  was 
omitted  during  the  recounting.  This  sweep  helps  to  pinpoint  gaps,  both  in  time  and 
events,  and  typically  aids  in  recall  of  the  missing  portions. 

Sweep  3  -  Expanding  on  the  incident  to  look  for  cues  and  factors  affecting 
teamwork:  The  interviewer  went  through  the  event  again,  this  time  probing  at  various 
points  and  asking  for  more  detailed  description  of  the  nontechnical  aspects  of  the 
situation.  This  sweep  involved  questioning  the  reasoning  process  and  looking  for  cues 
and  rationale  for  the  actions  taken  by  team  members. 

Sweep  4  -  “What  if’  queries:  The  interviewer  asks  questions  about  the 
participant’s  perceptions,  thoughts,  judgments,  and  actions,  and  what  would  have 
happened  if  aspects  of  the  scenario  had  been  different.  Each  question  is  designed  to 
extract  more  information  that  is  applicable  and  gain  a  better  understanding  of  the  story  as 
a  whole. 

c.  Interview  Transcription 

The  author  took  extensive  notes  in  addition  to  obtaining  an  audio 
recording  of  each  interview.  Rather  than  transcribing  the  whole  interview,  the  author  used 
the  notes  and  recordings  to  create  a  single,  full  report  of  the  experience.  The  events  were 
put  into  chronological  order  and  repetitions  were  omitted.  The  author  then  edited  the 
transcripts  into  a  single  format  that  was  concise  and  comparable  across  the  separate 
interviews.  The  author  then  pulled  significant  events  or  “statements”  from  each 
transcript.  These  statements  were  primarily  actions  that  the  OOD  took  while  standing  the 
watch.  Each  statement  would  be  observable  and  ratable. 
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3. 


Results 


Each  of  the  eight  participants  related  between  one  and  three  situations  from  their 
experience  for  a  total  of  16  scenarios.  After  transcribing  the  interviews,  the  author  was 
able  to  extract  149  separate  and  distinct  statements  from  the  transcripts  concerned  with 
the  nontechnical  skills  of  the  OOD.  Exemplar  statements  are  listed  below. 

•  “The  OOD  went  baek  to  the  bridge  wing  to  look  at  the  contact  again.” 

•  “The  OOD  reported  both  contacts  to  the  CO  prior  to  being  relieved.” 

•  “Once  the  ship  was  offieially  called  into  its  waiting  station,  the  OOD  gave 
the  order  to  proeeed.” 

•  “The  OOD  never  called  the  oiler  to  state  his  intentions.” 

•  “Once  the  ship  was  only  500  yards  off  the  oiler's  bow,  the  OOD  ordered 
the  Conning  Officer  to  mateh  the  oiler's  course  and  speed  to  avoid  a 
eollision.” 

•  “The  OOD  did  not  use  the  wind  envelope  manual  to  prepare  for  helicopter 
operations.” 

•  “The  OOD  continued  to  look  at  the  radar  scope  instead  of  looking  out  the 
windows  to  see  if  they  were  approaehing  the  oiler  too  fast.” 

•  “The  OOD  did  not  know  that  the  ship  was  four  miles  left  of  traek.” 

•  “The  OOD  ordered  the  bridge  team  to  stop  approaehing  the  other  vessel.” 

•  “The  OOD  told  the  XO  that  they  were  not  approaching  the  correct  vessel.” 

4.  Discussion 

The  scenarios  provided  ample  information  to  evaluate  the  validity  of  the 
nonteehnical  skills  taxonomy.  The  data  refleeted  real-life  situations  experieneed  by 
qualified  OODs  during  stressful  or  trying  times.  The  next  section  discusses  how  the 
statements  were  used  to  develop  a  valid  and  reliable  nontechnical  skills  taxonomy. 

E,  INTER-RATER  RELIABILITY  TESTING 

1,  Purpose 

The  author  tested  the  inter-rater  reliability  to  ensure  that  it  was  possible  for  two 
raters  to  reliably  use  the  system  to  classify  the  nonteehnical  skills  of  OODs. 
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2. 


Method 


Two  individuals  worked  together  to  test  the  inter-rater  reliability  of  the  skills 
taxonomy.  The  first  individual  was  the  author  who  is  a  Surfaee  Warfare  Offieer  and 
qualified  OOD.  The  seeond  partieipant  was  a  behavioral  psyehologist  with  experienee  in 
behavioral  markers.  He  has  produced  similar  products  for  other  fields  including  nuclear 
power  plant  control  rooms  and  U.S.  Navy  diving. 

The  inter-rater  reliability  testing  consisted  of  classifying  the  statements  extracted 
from  the  interviews  by  their  applicable  element  in  the  skills  taxonomy.  The  overall  goal 
was  for  both  raters  to  identify  each  statement  by  a  single  element  and  nontechnical  skill. 
If  each  statement  could  be  sorted  into  a  category  cleanly,  the  taxonomy  was  effective  and 
no  revisions  were  needed.  However,  if  certain  statements  did  not  fit  anywhere  or  applied 
to  multiple  categories,  the  system  needed  to  be  revised.  The  author  calculated  the 
effectiveness  of  the  inter-rater  reliability  using  Cohen’s  kappa. 

Cohen’s  kappa  (Cohen,  1960)  is  a  method  developed  to  calculate  the  inter-rater 
reliability  between  two  individuals.  It  is  considered  a  better  measurement  than  the  simple 
percentage  of  agreement  because  it  takes  chance  agreement  into  account.  Cohen’s  kappa 
(k)  produces  a  value  <  1.0.  The  closer  that  the  value  is  to  1.0,  the  better  that  the  inter-rater 
reliability  is  said  to  be.  Landis  and  Koch  (1977)  proposed  that  a  kappa  of  less  than  0  was 
indicative  of  poor  agreement,  between  0.0  and  0.20  indicates  a  slight  agreement;  between 
0.21  and  0.40  a  fair  agreement;  between  0.41  and  0.60  a  moderate  agreement;  between 
0.61  and  0.80  a  substantial  agreement;  and  between  0.81  and  1.00  almost  perfect 
agreement.  The  goal  of  this  study  was  to  produce  a  k  >  .81  between  the  raters.  As  will  be 
shown  below,  four  separate  iterations  of  ratings  were  required  before  an  acceptable  level 
of  kappa  could  be  achieved. 

a.  Iteration  1 

In  the  first  iteration,  the  two  raters  classified  25  randomly  selected 
statements  using  the  initial  taxonomy  (see  Table  2).  The  ratings  were  conducted 
independently.  The  first  iteration  resulted  in  an  inter-rater  reliability  of  k  =  .43  indicating 
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moderate  agreement.  While  this  was  a  surprisingly  good  result  for  the  first  run,  it  was 
elear  that  ehanges  needed  to  be  made  to  the  taxonomy  to  improve  the  inter-rater 
reliability. 


Every  nonteehnieal  skill  eategory  showed  some  degree  of  disagreement, 
espeeially  the  individual  elements  of  the  leadership  and  managing  stress  eategories.  There 
was  also  eonfusion  between  the  deeision  making,  situational  awareness,  and 
eommunieation  skill  eategories.  After  diseussion  of  areas  of  agreement  and  disagreement 
between  the  raters,  the  author  made  substantial  ehanges  to  the  taxonomy  with  the  aim  of 
improving  it.  Table  3  shows  the  revised  taxonomy. 


Nontechnical  Skill  Category 

Elements 

Leadership 

Managing  Workload 

Managing  Stress 

Decision  Making 

Considering  Options 

Balaneing  Risks 

Implementing  and  Reviewing  Deeisions 

Situational  Awareness 

Aetively  Gathering  Information 

Responding  to  Changes  in  Information 

Antieipating  Future  Events 

Communication 

Issuing  Orders 

Following  Orders  and  Proeedures 

Providing  Information 

Table  3.  OOD  Nonteehnieal  Skills  Taxonomy  Revision  1 

The  author  eliminated  the  “managing  stress”  nonteehnieal  skill  eategory 
and  added  it  as  an  element  under  leadership.  He  also  removed  the  two  unelear  elements 
from  leadership.  In  an  attempt  to  aehieve  some  elarity  between  the  other  eategories,  the 
author  ereated  new  elements  for  deeision  making  and  eommunieation  while  keeping 
situational  awareness  the  same. 

b.  Iteration  2 

After  revising  the  taxonomy,  the  raters  tested  it  again.  They  elassified  25 
different  statements  using  the  revised  taxonomy  (see  Table  3),  and  onee  again,  the  ratings 
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were  done  independently.  The  second  iteration  resulted  in  an  inter-rater  reliability  of  k  = 
.65  indicating  substantial  agreement.  The  improvement  was  expected,  but  it  was  apparent 
that  portions  of  the  system  were  still  unclear  and  difficult  to  use. 

The  leadership  category  showed  perfect  concurrence,  and  there  was  good 
agreement  on  the  communication  elements.  Unfortunately,  there  was  still  significant 
disagreement  between  decision  making  and  situational  awareness.  The  author  addressed 
this  problem  by  changing  the  elements  of  decision  making  because  they  were  the  least 
clear.  Table  4  shows  the  second  revision  of  the  taxonomy. 


Nontechnical  Skill  Category 

Elements 

Leadership 

Managing  Workload 

Managing  Stress 

Decision  Making 

Analytical  Decision  Making 

Following  Orders  and  Procedures 

Intuitive  Decision  Making 

Situational  Awareness 

Actively  Gathering  Information 

Responding  to  Changes  in  Information 

Anticipating  Future  Events 

Communication 

Issuing  Orders 

Providing  Information 

Receiving  Information 

Table  4.  OOD  Nontechnical  Skills  Taxonomy  Revision  2 

The  author  decided  that  broader  elements  would  be  easier  to  use  and  rate 
in  the  decision  making  category.  It  was  decided  to  use  the  three  naturalistic  decision 
making  styles  (analytical,  rule -based,  and  intuitive;  Klein,  2008).  However,  the  wording 
of  the  rule-based  decision  making  was  revised  to  better  fit  the  OOD  environment  by 
calling  it  “following  orders  and  procedures.”  Because  this  element  had  previously  been 
classified  under  communication,  the  author  also  had  to  make  minor  changes  to  that 
category. 
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c. 


Iteration  3 


The  raters  then  took  25  new  interview  statements  and  tested  the  reliability 
of  the  seeond  revision  of  the  taxonomy  shown  in  Table  4.  The  third  iteration  resulted  in 
an  inter-rater  reliability  of  k  =  .70  indieating  substantial  agreement.  While  there  were 
some  minor  improvements,  a  few  problems  still  existed  in  the  taxonomy. 

The  leadership  and  deeision  making  eategories  showed  very  good 
agreement.  However,  there  was  some  eonfusion  between  eommunication  and  situational 
awareness.  Speeifieally,  it  was  difficult  to  decide  whether  the  OOD  was  gathering 
information  or  receiving  information.  In  the  example  of  a  lookout  making  a  report,  the 
OOD  was  certainly  receiving  information,  but  he  was  also  improving  his  situational 
awareness.  The  author  decided  that  while  both  elements  could  apply  to  the  situation, 
gathering  information  in  the  situational  awareness  category  covers  a  wider  range  of 
events  and  is  more  useful.  Therefore,  the  “receiving  information”  element  was  eliminated 
from  the  communication  category.  The  situational  awareness  elements  also  were 
reworded  to  provide  further  clarification.  Table  5  shows  the  final  skills  taxonomy. 

d.  Iteration  4 

Due  to  the  relatively  minor  changes  made  to  the  taxonomy  in  the  third 
revision,  the  author  decided  to  rate  all  149  interview  statements  to  properly  test  the  latest 
version  of  the  taxonomy.  The  fourth  iteration  resulted  in  an  inter-rater  reliability  of  k  = 
.91  indicating  nearly  perfect  agreement.  As  the  level  of  inter-rater  reliability  was  higher 
than  0.81  (perfect  agreement  as  defined  by  Landis  &  Koch,  1977),  no  further  changes 
were  made  to  the  taxonomy. 

3.  Results 

Table  5  depicts  the  final  taxonomy  of  nontechnical  skills  and  the 
distribution  of  the  statements  across  the  skills  and  elements.  The  inter-rater  reliability 
testing  created  a  system  with  four  nontechnical  skill  categories:  leadership,  decision 
making,  communications,  and  situational  awareness.  The  following  sections  discuss  each 
category  along  with  the  associated  elements. 
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Skill 

Element 

Definition 

Example 

Skill 

% 

Element 

% 

within 

skill 

Overall 

element 

% 

Leadership 

Managing 

Watch  Team 

Effectively  setting  and  maintaining 
the  standards  of  the  watch  team. 

The  OOD  utilized  the  dead  time  in  the  schedule  to 
review  the  emergency  procedures  with  the 
helmsman. 

12.1 

44.4 

5.4 

Coping  with 
Stress 

Retaining  a  calm  demeanor  when 
under  pressure  and  demonstrating  to 
the  watch  that  one  is  under  control. 

Despite  the  added  pressure  from  the  XO,  the 
OOD  managed  the  stress  and  performed 
proficiently. 

55.6 

6.7 

Comms 

Providing 

Information 

Passing  information  along  to  other 
watch  stations  throughout  the  ship,  as 
well  as  other  assets  in  the  area. 

The  OOD  called  the  other  ships  in  formation  to 
inform  them  that  the  passing  oiler  was  dimly  lit 
and  difficult  to  see. 

28.2 

33.3 

9.4 

Issuing  Orders 

Effectively  giving  orders  to  other 
members  of  the  watch  team  and  other 
individuals  as  required. 

The  OOD  ordered  the  Engineering  Officer  of  the 
Watch  to  start  another  engine. 

66.7 

18.8 

Situational 

Awareness 

Gathering 

Awareness 

Actively  gathering  information  to 
keep  up  with  the  changing  situation. 

At  two  nautical  miles  out,  the  OOD  visually 
inspected  the  contact  through  binoculars. 

39.6 

66.1 

26.2 

Understanding 

Awareness 

Achieving  an  understanding  of  what 
the  available  information  means. 

The  OOD  identified  the  contact  as  a  fishing 
vessel  by  analyzing  the  lighting  configuration. 

25.4 

10.1 

Anticipating 
Future  Events 

Forward  planning  in  order  to 
anticipate  possible  future  problems. 

The  OOD  had  the  Conning  Officer  drive  slightly 
right  of  the  intended  track  because  he  knew  the 
wind  and  current  would  push  the  ship  to  the  left. 

8.5 

3.4 

Decision 

Making 

Analytical 

Decision 

Making 

Generating  and  comparing  multiple 
courses  of  actions  to  come  up  with 
the  optimal  solution. 

Once  the  oiler  was  located,  the  OOD  decided  to 
start  driving  towards  her  early  to  ensure  that 
there  was  plenty  of  time  to  set  up  later. 

20.1 

53.3 

10.7 

Following 

Orders  & 
Procedures 

Following  documented  procedures  or 
direct  orders  from  superior  officers. 

The  OOD  used  the  wind  envelope  guide  to  make 
sure  that  the  winds  were  sufficient  to  conduct 
flight  operations. 

40.0 

8.1 

Intuitive 

Decision 

Making 

Making  quick  decisions  based  upon 
prior  experience  and  intuition. 

When  the  mysterious  light  finally  materialized  as 
a  sailboat  300  yards  off  the  port  bow,  the  OOD 
immediately  ordered  “Hard  Right  Rudder.  ” 

6.7 

1.3 

Table  5.  Nontechnical  Skills  and  Elements  Percentage  Table 
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a. 


Leadership 


The  Navy  defines  leadership  as  the  “sum  of  those  qualities  of  intellect,  of 
human  understanding,  and  of  moral  character  that  enable  a  person  to  inspire  and  to 
manage  a  group  of  people  successfully”  (Stavridis  &  Girrier,  2007,  p.  8).  The  inter-rater 
reliability  testing  modified  the  original  three  elements  shown  in  Table  2  down  to  the  two 
shown  in  Table  5:  managing  watch  team  and  managing  stress. 

A  normal  underway  watch  can  be  characterized  by  long  stretches  of 
monotony  followed  by  periods  of  fast-paced  action.  Managing  the  workload  refers  to  the 
OOD’s  ability  keep  the  watch  team  functioning  through  the  peaks  and  valleys  of  activity. 

Life  at  sea  can  be  a  stressful  environment  for  Surface  Warfare  Officers. 
The  responsibilities  to  his  or  her  division  and  collateral  duties  pile  up,  and  fatigue  often 
becomes  an  issue.  The  ability  to  block  out,  or  compartmentalize,  the  stressors  and 
concentrate  on  the  watch  is  crucial  to  the  effective  performance  of  the  watch  team.  For 
this  reason,  managing  stress  is  a  crucial  element  in  the  leadership  of  the  OOD. 

b.  Decision  Making 

Decision  making  can  be  defined  as  “the  process  of  reaching  a  judgment  or 
choosing  an  option  to  meet  the  needs  of  a  given  situation”  (Flin  et  ah,  2008,  p.  41).  While 
the  focus  group  developed  three  broad  elements  of  decision  making,  the  elements  did  not 
allow  for  acceptable  levels  of  inter-rater  reliability.  As  a  result,  three  new  elements  were 
created  based  on  Klein  (2008):  analytical  decision  making,  following  orders  and 
procedures,  and  intuitive  decision  making. 

Analytical  decision  making  involves  the  comparisons  of  multiple  courses 
of  actions  to  develop  the  optimal  solution.  The  OOD  encounters  many  situations  that 
require  a  careful  analysis  of  multiple  options.  The  ability  to  consistently  select  the  best 
alternative  is  a  valuable  skill  for  an  OOD. 

The  OOD  deals  with  many  situations  that  are  highly  structured  through 
either  documented  procedures  or  direct  orders  from  superior  officers.  When  properly 
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utilized,  night  orders,  cheeklists,  and  emergeney  operating  proeedures  are  a  few  examples 
of  the  tools  that  make  the  OOD’s  job  mueh  easier.  The  OODs  desire  and  ability  to  take 
advantage  of  these  tools  is  an  important  element  of  deeision  making. 

Intuitive  deeision  making  refers  to  situations  where  there  is  insuffieient 
time  to  follow  a  doeumented  proeedure  or  eonduct  an  analysis  of  alternatives.  These 
instanees  require  the  OOD  to  make  quiek  deeisions  based  upon  prior  experienee  and 
intuition.  These  situations  tend  to  be  the  most  eritical  and  the  CO  must  have  eonfidenee 
that  the  OOD  will  be  able  to  make  the  right  decision. 

c.  Situational  Awareness 

Situational  awareness  is  “the  perception  of  the  elements  in  the 
environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and 
the  projection  of  their  status  in  the  near  future”  (Endsley,  1995,  p.  36).  As  discussed 
earlier,  the  focus  group  created  three  elements  for  situational  awareness  based  upon  the 
levels  of  Endsley’s  (1995)  model  of  situation  awareness.  These  three  elements  survived 
the  inter-rater  reliability  testing  with  only  minor  changes  to  the  wording.  The  new 
elements  are  gathering  awareness,  understanding  awareness,  and  anticipating  future 
events. 

The  environment  around  a  naval  vessel  is  constantly  in  flux.  The  OOD 
must  actively  gather  awareness  about  his  or  her  surroundings  to  keep  up  with  the  changes 
and  adjust  accordingly.  The  OOD  has  many  tools  available  to  help  increase  situational 
awareness  including  radars,  lookouts,  and  his  or  her  own  eyes.  Properly  utilizing  these 
sensors  and  tools  is  crucial  to  gathering  accurate  awareness  about  the  surrounding 
environment. 

Simply  gathering  information  is  not  enough  to  achieve  good  situational 
awareness.  The  OOD  needs  to  use  the  tools  at  his  or  her  disposal  to  achieve  an 
understanding  of  what  the  information  means. 
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Once  there  is  a  general  understanding  of  the  situation,  effective  OODs  will 
be  able  to  projeet  the  information  into  the  future  in  order  to  antieipate  what  is  eoming 
next.  Anticipating  future  events  is  erueial  to  keeping  the  ship  out  of  dangerous 
environments  and  situations. 

d.  Communication 

Communieation  is  “the  exehange  of  information,  feedbaek  or  response, 
ideas,  and  feelings”  (Flin  et  ah,  2008,  p.  69).  The  inter-rater  reliability  testing  proved  that 
the  original  elements  of  eommunieation  ereated  by  the  foeus  group  were  largely 
ineffeetive.  The  new  elements  were:  providing  information  and  issuing  orders. 

The  OOD  is  directly  responsible  for  providing  information  to  the  CO  and 
XO  as  well  as  the  navigator  in  eertain  situations.  He  or  she  must  pass  information  along 
to  other  assets  in  the  area,  as  well  as  to  other  wateh  stations  throughout  the  ship.  The 
ability  to  provide  the  information  in  a  elear  and  eoneise  manner  greatly  inereases  the 
effeetiveness  of  the  OOD  and  the  eonfidenee  of  the  ehain  of  eommand. 

The  OOD  is  direetly  in  eharge  of  a  wateh  team  eonsisting  of  at  least  eight 
sailors.  The  engineering  and  eombat  watehes  also  report  to  the  OOD  during  normal 
operations.  These  responsibilities  neeessitate  that  the  OOD  issues  orders  to  a  wide  range 
of  individuals  in  many  different  situations.  Therefore,  issuing  orders  is  an  important  skill 
for  the  OOD. 

4.  Discussion 

Examining  the  NTSOD  skills  and  elements  that  were  most  eommonly  used  to 
categorize  the  interview  statements  provides  evidenee  as  to  the  nonteehnieal  skills  that 
should  be  emphasized  during  OOD  training.  From  Table  5,  it  ean  be  seen  that  the  most 
frequently  used  elements  were  ‘issuing  orders’  and  ‘gathering  information’.  These  two 
elements  were  used  to  eategorize  almost  half  of  the  interview  statements. 
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Assertiveness  and  giving  direet  and  explieit  orders  would  appear  to  be  an 
important  nonteehnieal  skill  that  should  be  emphasized  in  training  junior  OODs.  This 
finding  is  not  unexpected  given  that  the  OOD  is  the  leader  of  the  bridge  team  and  must 
provide  direction  to  the  13  different  watch  stations  that  report  to  him  or  her. 

Gathering  information  was  the  most  commonly  used  NTSOD  element  in  this 
study.  It  was  used  to  classify  26.2%  of  the  interview  statements.  The  failure  of  U.S. 
Navy  sailors  to  gather  information  was  a  key  causal  factor  in  the  ramming  of  the  Spanish 
freighter  by  the  USS  Eisenhower  (National  Transportation  Safety  Board,  1990).  In 
Endsley’s  (1995)  three  stage  model  of  situational  awareness,  the  first  stage  of  gathering 
information  is  where  most  errors  occur.  Jones  and  Endsley  (1996)  examined  situational 
awareness  errors  in  civil  aviation,  while  Sneddon,  Mearns,  and  Elin  (2006)  examined 
them  in  offshore  oil  drilling.  It  was  found  that  the  most  situational  awareness  errors 
occurred  while  gathering  information  (76.3%  and  67.0%  respectively),  as  opposed  to 
understanding  the  meaning  of  the  information  (20.3%  and  20.0%  of  errors  respectively), 
or  anticipating  future  states  (3.4%  and  13.0%  of  errors  respectively).  Therefore,  given  the 
importance  to  the  OOD  of  the  skill  of  gathering  information,  in  combination  with  it  being 
where  most  errors  occur,  the  OOD  must  be  given  training  that  emphasizes  and  provides 
practice  in  effective  information  gathering  (e.g.,  maintaining  a  good  scan;  see  Elin  et  al, 
2008,  for  more  details). 

Intuitive  decision  making  was  rarely  used  to  classify  the  interview  statements  (it 
was  only  used  to  classify  1.3%  of  the  statements).  Part  of  the  explanation  for  this 
surprising  finding  may  be  that  the  operations  of  a  U.S.  Navy  ship  are  highly 
proceduralized.  However,  although  it  may  be  rare,  the  ability  to  make  intuitive  decisions 
is  crucial  for  the  OOD  in  risky,  fast  moving,  or  non-normal  situations  for  which  there  is 
no  procedure. 
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F.  PROTOTYPE  BEHAVIORAL  MARKER  SYSTEM 

1,  Purpose 

The  final  stage  in  the  development  of  the  Nonteehnical  Skills  for  Officers  of  the 
Deck  (NTSOD)  rating  system  was  to  use  the  categories  and  elements  shown  in  Table  5  to 
develop  a  nontechnical  skills  rating  form  that  could  be  used  by  the  fleet. 

2,  Method 

In  order  to  turn  the  nontechnical  skills  taxonomy  into  a  rating  form,  the  author 
studied  multiple  behavioral  marker  systems.  Each  had  its  strengths,  but  the  format  of  the 
NOTSS  system  (Figure  6)  showed  the  most  potential  for  use  in  the  Surface  Navy 
environment  and  had  received  acceptable  usability  ratings  from  surgeons  (Yule  et  al, 
2006).  The  NOTSS  system  employs  a  simple  format  similar  to  that  used  in  other 
behavioral  marker  systems.  The  header  section  allows  the  assessor  to  personalize  each 
form  to  the  individual  being  assessed  and  the  particular  evolution  that  they  are 
performing.  Another  useful  feature  of  the  NOTSS  system  is  the  amount  of  space 
available  for  taking  notes.  The  notes  enabled  by  this  space  allow  the  assessor  to  provide  a 
higher  level  of  feedback  to  the  trainee. 

The  NOTSS  behavioral  marker  system  was  developed  for  surgeons,  so  it  did  need 
to  be  adapted  before  it  could  be  used  to  assess  OODs.  The  author  changed  the 
information  in  the  header  section  to  apply  to  standing  watch.  He  included  areas  to  write 
the  ship  name,  trainee  and  assessor  name,  and  the  date.  The  author  also  felt  that  it  was 
important  to  include  spaces  for  the  watch  (2200-0200,  0200-0700,  etc.)  and  evolutions 
(underway  replenishment,  normal  underway  steaming,  etc.). 

The  author  chose  to  use  a  four-point  (unsatisfactory,  marginal,  satisfactory, 
outstanding)  rating  scale.  The  four-point  scale  allows  for  a  higher  degree  of 
discrimination  than  a  two-  or  three-point  scale.  The  even  number  also  forces  the  assessor 
to  make  a  judgment  about  the  trainee’s  performance.  There  will  be  times  when  not  every 
skill  is  represented  during  a  watch.  Therefore,  the  author  did  include  the  choice  of  not 
observed  (N/0). 
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The  author  created  a  body  similar  to  the  NOTSS  system  that  allows  the  assessor 
to  give  scores  for  both  the  individual  elements  and  the  nontechnical  skills  as  a  whole. 
This  structure  allows  the  trainee  to  get  feedback  regarding  the  individual  skills  that  need 
improvement,  but  it  also  allows  the  CO  or  XO  to  get  a  quick  idea  of  the  performance  by 
just  looking  at  the  broader  categories.  The  author  called  the  completed  behavioral  marker 
system  the  Nontechnical  Skills  for  Officers  of  the  Deck  (NTSOD)  rating  form.  Figure  8 
depicts  the  proposed  NTSOD  rating  form.  Detailed  instructions  on  the  use  of  the  NTSOD 
system  are  not  provided  here,  but  can  be  found  in  Long,  O’Connor,  &  McCauley  (2010). 

3,  Results 

Figure  8  shows  the  completed  NTSOD  Rating  Form. 
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Ship 

Trainee 

Watch 

Date 

Assessor 

Evolution 

Categoiy 

Categoiy 

Rating* 

Element 

Element 

Rating* 

Notes 

Leadenhip 

Managing  Watch  Team 

Coping  wnh  Stress 

Commimicatioas 

ftovTdmg  Infonmtion 

Issuing  Orders 

SituatioDal 

Attmeness 

Gathering  Awareness 

Understanding  Awareness 

Antic^iating  Future  Ev'ents 

Decision  Making 

Anahtjcal  Decision  Making 

FoDowmg  Orders  &  Procedures 

Inturtrve  Decision  Making 

*  1  -  Unsatisfector)-,  2  -  Marginal;  3  -  SaQsfactor\'  4  -  Outstanding;  N  O  -  Not  Obsmied 


1  -  Unsatisfectorv';  Watchstander  could  endanger  ship  and  crew  without  considerable  unpro\Tement 

2  -  Marginal:  Watchstanda  requires  improvement. 

3  -  Satisfectors'  Watchstander  perform^  at  an  acceptable  les'el,  but  room  for  m^ovement  exists 

4  -  Outstanding:  Watchstander  performed  at  a  consistently  high  le\  el 

N  O  -  Not  Observed:  Element  or  skill  was  not  observed  durmg  this  evolution 

Figure  8.  Nontechnical  Skills  for  Officers  of  the  Deck  (NTSOD)  Rating  Form 
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4. 


Discussion 


The  prototype  Nonteehnical  Skills  for  Offieers  of  the  Deck  (NTSOD)  taxonomy, 
shown  in  Figure  8,  provides  a  format  for  evaluating  the  nontechnical  skills  of  OODs  that 
was  based  upon  a  sound  research  foundation.  However,  further  developmental  work  is 
required  to  ensure  that  the  system  can  be  reliably  used  by  evaluators.  Once  a  prototype 
behavioral  marker  system  has  been  developed,  the  next  stage  is  for  representative  end 
users  to  rate  standardized  video  enactments  of  scenarios  similar  to  those  encountered  in 
the  actual  environment  (Fletcher  et  ah,  2004;  Yule  et  ah,  2008).  The  purpose  of  this 
evaluation  is  to  assess  the  sensitivity,  inter-rater  reliability,  and  internal  reliability  of  the 
system.  The  scenarios  are  typically  filmed  in  a  simulator  with  subject  matter  experts 
acting  the  main  roles. 

Should  an  acceptable  level  of  reliability  and  sensitivity  be  achieved,  the  usability 
of  the  behavioral  marker  system  is  then  evaluated  by  evaluating  the  behaviors  of  trainees 
in  the  actual  environment  (i.e.,  flight  deck,  operating  theatre,  ship’s  bridge,  etc.). 
Completed  rating  forms  are  examined  to  assess  whether  all  the  elements  and  categories 
are  being  used  and  information  is  obtained  from  the  raters  on  the  usability  of  the 
behavioral  marker  system. 

The  NTSOD  taxonomy  not  only  has  implications  for  the  evaluation  of  the 
nontechnical  skills  of  OODs,  but  could  also  be  used  to  provide  a  research  foundation  for 
the  content  of  the  U.S.  Navy’s  bridge  resource  management  (BRM)  program.  As  stated 
by  Hetherington  et  al.  (2004)  “a  review  of  the  literature  reveals  that  there  appears  to  be 
no  empirical  foundation  for  this  type  of  course  beyond  research  that  was  originally 
conducted  in  the  formation  of  aviation  [crew  resource  management]_CRM  courses”  (p. 
407).  The  U.S.  Navy’s  BRM  program  is  no  exception.  Carter-Trahan  (2009)  carried  out  a 
study  of  the  attitudes  and  knowledge  of  OODs  regarding  human  factors  issues  that  have 
been  identified  as  causal  to  mishaps  in  high-risk  organization.  She  stated  that  there  was  a 
need  for  a  systematic  research  effort  to  identify  the  particular  human  factors  issues  that 
should  be  included  as  part  of  the  Navy’s  BRM  program. 
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The  first  stage  in  the  development  of  any  training  program  should  be  a  needs 
assessment.  In  the  eontext  of  nonteehnieal  skills  training,  sueh  as  BRM/CRM,  a  training 
needs  assessment  is  neeessary  to  identity  the  skills  to  be  trained  (Flin  et  al,  2008).  To 
illustrate,  extensive  baekground  research  was  carried  out  to  identify  the  seven  skills  that 
form  the  basis  of  U.S.  Naval  aviation  CRM  training  (decision  making,  assertiveness, 
mission  analysis,  communication,  leadership,  adaptability/  flexibility,  and  situational 
awareness;  Prince  &  Salas,  1993).  The  needs  assessment  ensured  that  the  training  was 
based  upon  a  sound  research  foundation  and  that  limited  time  and  resources  for  training 
were  being  effectively  utilized. 

G.  CONCLUSION 

To  identity  the  applicable  nontechnical  skills  for  OODs,  the  author  used  a 
combination  of  literature  review,  focus  groups,  and  interviews.  Statements  extracted  from 
the  interviews  provided  the  tools  that  were  necessary  to  test  the  reliability  of  the 
taxonomy  when  applied  to  actual  surface  Navy  situations.  Testing  the  inter-rater 
agreement  multiple  times  shaped  the  list  of  skills  into  a  much  more  effective  and  usable 
tool.  The  next  chapter  discusses  how  the  author  used  the  nontechnical  skills  taxonomy  to 
create  a  rating  form  that  could  be  used  to  assess  the  abilities  of  OODs. 


53 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


54 


V.  SUMMARY,  RECOMMENDATIONS,  AND  CONCLUSION 

A.  SUMMARY 

This  thesis  started  with  the  goal  of  ereating  a  behavioral  marker  system  that  could 
be  used  for  assessing  the  nontechnical  skills  of  OODs.  The  literature  clearly  indicates  that 
human  error  is  a  leading  cause  a  major  mishaps,  and  the  lack  of  nontechnical  skills  is  a 
leading  cause  of  human  error.  This  situation  is  especially  true  in  high-risk  organizations 
such  as  the  Surface  Navy.  Because  it  is  impossible  to  create  a  system  capable  of 
addressing  an  entire  ship,  the  OOD  was  the  logical  place  to  begin. 

The  author  used  a  combination  of  literature  review,  focus  groups,  interviews,  and 
inter-rater  reliability  testing  to  identify  the  nontechnical  skills  that  are  applicable  to  the 
OOD  watch  station.  Once  the  skills  were  identified,  the  author  used  the  taxonomy  to 
create  the  Nontechnical  Skills  for  Officers  of  the  Deck  (NTSOD)  rating  form.  The 
NTSOD  system  can  be  used  to  observe  and  assess  the  nontechnical  skills  of  OODs  and 
candidates  trying  to  complete  the  qualification.  The  author  believes  that  with  consistent 
use,  the  NTSOD  system  can  increase  the  effectiveness  and  safety  of  OODs. 

B,  RECOMMENDATIONS  FOR  FURTHER  RESEARCH 

1.  Recommendation  1 

Additional  research  should  be  conducted  to  test  the  NTSOD  system’s  reliability 
and  usability  while  being  used  in  the  fleet  or  simulators.  This  research  could  support  the 
findings  that  the  NTSOD  system  accurately  assesses  the  nontechnical  skills  of  OODs. 
The  research  could  also  investigate  the  effects  that  the  NTSOD  system  has  on  the  overall 
performance  of  OODs. 

2.  Recommendation  2 

The  results  of  the  present  study  should  be  used  to  provide  a  research  foundation  to 
Bridge  Resource  Management  (BRM)  training  so  that  it  more  effectively  addresses  the 
skills  required  to  safely  stand  the  watch. 
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3. 


Recommendation  3 


While  the  OOD  is  the  most  important  wateh  station  onboard  surfaee  ships,  there 
are  many  other  important  positions  sueh  as  Taetieal  Aetion  Offieer  (TAO)  and 
Engineering  Officer  of  the  Watch  (EOOW).  Similar  watches  also  exist  on  submarines 
and  merchant  marine  shipping.  The  methods  described  in  this  thesis  should  be  used  to 
develop  behavioral  marker  systems  for  addressing  the  nontechnical  skills  of  those  watch 
stations  as  well. 

C.  CONCLUSION 

Hetherington  et  al  (2006)  stated  that  there  are  many  gaps  in  the  maritime  human 
factors  literature.  The  NTSOD  taxonomy  is  a  step  towards  filling  these  gaps.  Although 
further  research  is  required  to  establish  the  reliability  and  validity  of  the  NTSOD 
taxonomy,  it  is  one  of  the  first  research-based  behavioral  marker  systems  developed  for 
use  in  a  maritime  environment. 

Human  error  can  never  be  eliminated.  However,  identifying,  training,  and  giving 
feedback  on  the  nontechnical  skills  that  are  required  for  safe  and  effective  performance 
will  ensure  that  sailors  have  the  appropriate  skills  for  minimizing,  detecting,  and 
mitigating  error  before  it  leads  to  an  accident. 
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